How fast can your GPU actually run a transformer?

A playable roofline model — pick a GPU, shape a transformer, and watch where the FLOPs go.

Draft stub — the real essay goes here. The point of this post in the skeleton is to prove that prose and live computation can share a page.

Everyone quotes peak TFLOPs. Almost nobody hits them. During decode, a transformer reads every weight once per token step — so until your batch is large enough, the GPU spends its life waiting on HBM, not multiplying matrices.

Find the ridge

Try it. Watch the “bound by” readout flip as you push the batch size past the ridge:

Decode roofline / bench 01

[MEMORY-BOUND · 5.2% MFU]

GPU d_model 4096 layers 32 batch 8

params 6.4B

weights (bf16) 13 GB

decode speed 1.3k tok/s

FLOPs utilization 5.2%

bound by memory bandwidth (ridge at batch ≈ 153)

Simplified model: dense weights read once per step, no KV cache, no kernel overhead. Directionally honest, numerically optimistic.

Read the result

Two things to notice:

The ridge point is a property of the GPU, not the model. Peak FLOPs divided by memory bandwidth. On an A100 that’s ~153: below that batch size, you are paying for silicon you cannot feed.
Utilization is a choice you make at design time — d_model, layer count, and batch geometry decide your ceiling before a single kernel is written.

The full essay will add the KV cache (which makes everything worse), quantization (which moves the ridge), and real measured numbers against this toy model.