How this works

What makes two results comparable

Results share a rank only when every one of these keys matches. Nothing else is ranked together, no matter how similar it looks.

category · tier · lane · profile_version · runtime · runtime_version · gpu_count · parallelism

The profile file that defines those rules is hashed, and its hash travels inside every manifest. If a profile were ever edited after release, results produced before and after would carry different hashes and would stop sharing a leaderboard automatically. Released profiles are immutable; a change publishes a new version instead.

Why three tiers

LocalMax targets everything from a 12 GB RTX 3060 to a dual RTX PRO 6000 workstation at 192 GB and a DGX Spark at 128 GB of unified memory. That is a sixteenfold range in capacity and roughly sixfold in memory bandwidth.

A single small model across that range measures almost nothing but memory bandwidth. The second RTX PRO 6000 would sit idle, and DGX Spark — whose entire value is capacity — would rank below an RTX 3060. So a tier is defined by the VRAM its ranked run is meant to fill: 12 GB, 24 GB, 64 GB. Each has its own fixed model.

The Frontier baseline is a 32B at BF16, roughly 64 GB. That is deliberate: it fits both a 128 GB Spark and a 2×96 GB workstation. A 70B BF16 baseline would have excluded Spark from the tier that exists to describe it.

Why three quantization lanes

Precision support is not uniform across the hardware here. Ampere has no FP8 unit at all; FP4 is Blackwell only. Pinning one precision would either exclude the RTX 3090 — still one of the most common cards in local AI — or make Blackwell's headline advantage invisible.

• BF16 runs everywhere. This is the fair cross-generation baseline and the lane that spans the most hardware.
• INT4 (AWQ) runs on Ampere and up, and fits a larger model in the same VRAM. It is what most people actually run locally.
• NVFP4 is Blackwell only: RTX 5090, RTX PRO 6000, DGX Spark.

Each lane is its own leaderboard. Ranking them together would compare different numerical workloads and quietly reward whoever picked the loosest precision. A fourth FP8 lane is reserved in the schema but ships no profile in v1.

What is measured

Decode and prefill throughput are reported separately. This is the single most informative split on the site. Decode tracks memory bandwidth; prefill tracks compute. Reporting only time-to-first-token would conflate prefill with scheduling overhead and hide the real difference between a bandwidth-rich RTX 5090 and a capacity-rich DGX Spark.

LLM profiles are ranked on decode throughput, behind gates on TTFT p95 and inter-token latency p95: a system that cannot answer interactively is published but not ranked. Vision is ranked on images per minute, behind a deterministic answer-accuracy gate — a fast but wrong system does not get a rank. Diffusion is ranked on seconds per denoising step, which is resolution- and step-count-normalised; images per minute is derived from it and is what the headline shows.

Diffusion is compute-bound where the LLM profiles are bandwidth-bound. That is on purpose. Without it, three categories would produce one memory-bandwidth chart in three costumes.

Power is typed, not just measured

A discrete card reports board power. A GB10 reports power at the SoC module, where CPU, GPU and LPDDR share a rail. These are different physical quantities, and averaging them into one efficiency score would be meaningless.

So every result carries a power domain — gpu_board,soc_module, or unavailable — and results are never ranked for energy across domains. Unified-memory wattage is shown, labelled, and excluded from the efficiency view.

Verification states

Verified

Official signed container digest, frozen profile whose hash matches, every required artifact present, telemetry covering at least 99% of the measured run, and every derived metric recomputed from the raw records within tolerance. Ranked.

Community

Valid manifest and signature, but incomplete evidence or an unofficial image. Every raw value is published; the result is never ranked.

Flagged

A value sits outside the plausible range for the hardware. Held for human review. Nothing is altered — a suspicious number stays exactly as submitted, with the finding attached.

Certified — an independent operator repeating the run — is deferred until the protocol has settled.

How a result is validated

1. Schema validation of the full manifest against a published JSON Schema.
2. Ed25519 signature check over the RFC 8785 canonical form of the manifest.
3. Profile hash comparison against the published profile bytes.
4. Runtime flags compared against the pinned set, flag by flag.
5. Hardware eligibility: total VRAM against the tier, architecture against the lane, GPU count against the profile.
6. Evidence retrieval: every artifact must exist in the store at the exact declared size, having been verified against its SHA-256 during upload.
7. Secret scan over text evidence. A hit rejects the submission and purges the objects.
8. Metric recomputation. Percentiles, throughput, error rates and accuracy are recalculated from the raw per-request records and compared to the declared values within tolerance.
9. Plausibility bounds per profile. A violation flags for review; it never rewrites a value.
10. Ranking gates, then duplicate detection against the submitting system's history.

This runs asynchronously on a queue rather than inline, which is why submission returns immediately even under load.

What this cannot tell you

A signature proves a bundle came from one runner installation and has not been modified. It does not prove the reported hardware is real, and it does not prove the operator was honest. Verified describes protocol compliance and evidence consistency — it is not an audit.

What actually protects the data is that a fabricated number would also require a self-consistent set of raw per-request records, a telemetry trace, and a value that sits plausibly within the distribution for that GPU. All of which is public, so anyone can check.

Finally: these are inference performance measurements under one fixed workload. They say nothing about model quality, nothing about how a model behaves on your prompts, and nothing about performance under a workload shaped differently from ours.