AI Compute Reference

What is an AI compute hardware reference?#

When you read that a GPU delivers “1979 TFLOPS” or a chip has “192 GB of memory,” those numbers only mean something once you know the rules behind them: which numeric precision the figure was measured at, whether it includes structured sparsity, whether it is floating-point throughput (TFLOPS) or integer throughput (TOPS), and what bandwidth the memory actually supplies. Vendors publish headline numbers in their most flattering configuration; without a reference that states the conditions, two chips cannot be compared honestly.

This page is that reference. It gathers four things an ML engineer or infrastructure buyer actually needs, side by side: the compute-unit glossary (what FLOPS, TFLOPS, TOPS, and MAC actually mean), the numeric precision formats (FP64 down to INT4, in typical throughput order), the accelerator types (CPU, GPU, TPU, NPU, LPU, MAU), and a hardware table of specific chips with their memory capacity, memory bandwidth, and dense peak throughput at FP16, FP8, and INT8 — each figure verified against the vendor datasheet, with the exact caveat noted when a number could mislead.

How to use it#

1. Search across everything. The single search box at the top filters all four sections at once by name, vendor, process node, or spec — typing H100, NVIDIA, 192gb, or 4.8tb/s narrows the hardware table.
2. Read the four sections.
   • Compute units — the SI-prefixed symbols (FLOP up to EFLOPS, plus TOPS and MAC) with their base-10 exponents, so you can convert between them.
   • Precision formats — FP64, FP32, TF32, FP16, BF16, FP8, INT8, INT4, each with its bit width and typical use case, ordered roughly by rising throughput on a modern tensor core.
   • Accelerator types — the one-line distinction between a CPU, GPU, TPU, NPU, LPU, and MAU.
   • Hardware table — name, vendor, process, memory (GB), memory bandwidth (TB/s), and dense FP16 / FP8 / INT8 throughput, with a caveat note per chip.
3. Trust the blanks. Where a cell is empty rather than zero, the figure could not be verified as a dense peak — the note explains why (AMD’s MI300X publishes sparse peaks, so dense is roughly half). It is more honest to leave it blank than to print a number with a hidden condition.

Key features#

• Dense peaks only, with sparsity called out. NVIDIA figures are dense (without 2:4 structured sparsity); the note reminds you that their headline marketing numbers often double when sparsity is turned on. No silent “with sparsity” inflation.
• TFLOPS and TOPS kept apart. The glossary states plainly that TOPS (the OPS family) is not directly comparable to TFLOPS (the FLOPS family) — a confusion vendors happily exploit.
• All four precisions in throughput order. See at a glance that FP8 roughly doubles FP16, and INT8 roughly doubles BF16, on a given architecture — the economic reason quantization exists.
• Memory and bandwidth, not just compute. For large-language-model inference, memory capacity and bandwidth often matter more than peak FLOPS, so they are first-class columns.
• Sources cited by name, not link. Every figure names the datasheet or product page it came from, with no outbound URL, so you can verify against the vendor’s own documentation.

Worked example#

Compare the NVIDIA line across three generations using the hardware table. Search NVIDIA and you get the A100, H100, and H200 rows together:

Two things jump out. First, the H100 to H200 jump added no compute — both share the GH100 compute die; the H200 is a memory refresh to HBM3e, nearly doubling capacity (80 → 141 GB) and bandwidth (3.35 → 4.8 TB/s). For an LLM inference workload that is memory-bound, that is a bigger real-world win than more FLOPS would have been. Second, the A100 has no FP8 column — FP8 is a Hopper-generation feature, so pre-Hopper chips simply do not have it.

Then pull up the AMD and Google rows. The MI300X shows the largest memory (192 GB) and bandwidth (5.3 TB/s) of any chip listed, but its FP16/FP8/INT8 cells are intentionally blank, with the note explaining AMD publishes structured-sparse peaks (1307 FP16 sparse, 2614 INT8 sparse) and dense is roughly half — printing the sparse number next to NVIDIA’s dense one would be a misleading comparison. The TPU rows show a different convention again: TPU v4 reports BF16 equal to INT8 at 275 per chip, while the cost-optimized v5e doubles INT8 to 393 against 197 BF16.

FAQ#

TFLOPS versus TOPS — are they the same thing?#

No, and confusing them is the most common spec-sheet trap. TFLOPS measures floating-point operations per second (the FLOPS family, base-10 prefixes: MFLOPS, GFLOPS, TFLOPS, PFLOPS). TOPS measures integer or generic operations per second (the OPS family) and is what you usually see quoted for INT8 quantized inference. Because the underlying operation differs, a chip’s TOPS number is not directly comparable to its TFLOPS number — and a MAC (one multiply-accumulate) is conventionally counted as two FLOPs, which is part of why vendor TOPS figures sometimes appear to double-count.

Why are the MI300X compute cells blank?#

Because AMD publishes its peak throughput with structured sparsity enabled (for example, 1307 FP16 TFLOPS sparse), whereas this table reports dense peaks to keep every chip comparable. Dense MI300X is roughly half of the sparse figure, but a clean, verified dense number is not consistently available, so the cell is left blank rather than printing a number with a hidden “sparse” condition. The memory and bandwidth — 192 GB and 5.3 TB/s — are verified and shown.

What is the difference between FP16 and BF16?#

Both are 16 bits, but they spend those bits differently. FP16 has more precision but a narrow numeric range; BF16 trades precision for the same range as FP32. That range matters for training, where gradients can vary wildly in magnitude, which is why BF16 has become the default training format on modern tensor cores even though it and FP16 occupy the same bit width. The precision-format section lists both with their typical use.

What does “peak” mean — will my model actually hit these numbers?#

Almost never. These are theoretical peaks — the throughput the chip could sustain if every cycle were perfectly utilized with no memory stalls, which real workloads never achieve. Sustained throughput is always lower and depends on the kernel, the model architecture, batch size, and whether the workload is memory- or compute-bound. Treat the peak as an upper bound for comparing architectures, not as a prediction of your model’s speed.