On composition in memory-bound LLM inference
I.
With published numbers one can set up the following calculation. An H100 has TB/s of memory bandwidth. Quantization cuts the bytes moved per weight to a quarter. Speculative decoding produces several tokens from a single read of the weights. Dynamic sparsity never fetches most of the weights at all. Multiply the three factors, , and you get twenty-four fold, that is, TB/s of effective bandwidth. I set up this calculation and for a while I believed it. Each input came from a separate paper, and each paper was right about its own measurement. What was wrong was the multiplication itself.
A speedup factor is not a property of a method; it is a ratio defined against a particular baseline. Quantization's four-fold gain is measured against byte traffic in which the weights are read as FP16. Sparsity's three-fold gain is measured against traffic in which all of the weights are fetched. Apply the two together and the baseline against which each ratio was defined no longer exists; the published factors no longer apply. That is why the multiplication is wrong: it holds two mutually exclusive reference states in the same equation.
This is not arithmetic sloppiness but a consequence of resource sharing. All three mechanisms use the same channel from HBM to SRAM for every generated token. When two methods that profit from the same channel are stacked, the second cannot take again the share the first has already taken. The gains do not compound in full; the composition is sub-multiplicative.
II. The Mechanics of Interference
The concrete case is the composition of speculative decoding with sparsity. Sparsity's gain rests on roughly seventy percent of the weights never being fetched from HBM for a single position. As the draft tree deepens, a single read of the weights serves several positions, but the active neuron sets of those positions do not coincide. The set that must be read is the union of the individual sets, and with depth it approaches the full weight matrix. As the keep rate goes to one, the router's cost stays fixed. What sparsity earns on its own it loses inside speculation; in a sufficiently deep tree it turns into a net loss.
The reverse is also possible, and it matters for not mistaking interference for a one-way tax. Draft positions are neighboring steps of the same context; their active neuron sets overlap more than two randomly chosen sets would. If the overlap is high enough, the union grows slowly and sparsity still leaves a margin inside a shallow tree. The composite of the two mechanisms is neither a product nor zero; it is a value that depends on the overlap rate and has to be measured.
The second source of interference is regime dependence. The composition of memory traffic changes with context length. At short context most of the bytes read are weights, and weight quantization pays directly. As the context grows, KV-cache traffic overtakes weight traffic; for a 70B model the crossover sits on the order of tens of thousands of tokens. On the far side of the crossover the same quantization factor applies to a shrinking share of the traffic, and the number drops. The latency-oriented single-sample regime and the throughput-oriented batched regime split the same way: with batching, arithmetic intensity rises, the bottleneck can stop being memory, and every factor on the memory side fades together. A factor is measured in one regime; reported without declaring the regime, it reads as a universal property of the method.
III. Three Flaws of the Measurement Regime
Nobody in this picture is measuring wrongly. Most of the pairwise interactions are already in the literature: the compatibility limits of quantization with speculation, union growth under batching, and sparsity co-designed with the memory layout have each been published separately. What is missing is not individual findings but the accounting that gathers them into one ledger. The problem knots in three places in how results are reported.
The first is the isolated ablation. An ablation table is single-variable: one mechanism is switched on, the others are held fixed, the gain is written down. The interaction term is never measured in this design; the row in which two mechanisms are on together is absent from most tables. Composition is thereby left to assumption rather than data, and the default assumption is the product, because the product is the easiest thing to compute.
The second is undeclared regimes. Batch size, context length, which side of the KV crossover the measurement sits on, and whether the target is latency or throughput determine a factor's domain of validity. These facts usually live in the experimental setup but are not attached to the headline number. Once the number circulates on its own, the domain of validity stays behind.
The third is baseline favoritism. Picking the comparison reference from whatever is at hand usually means picking the one that favors you. Using the hardware's sparsity-assuming catalog value as peak compute is the typical case; draw the roofline with that value and every gain on the memory side looks larger than it is. I made this mistake in my own accounting and did not notice it until I corrected it. Favoritism does not arise from intent; it arises from taking the default value without questioning it.
IV. Three Reporting Norms
The remedy for the three flaws is three norms, and all three ask for extra honesty, not extra experiments.
First, report the interaction term. If and are used together, the table must contain three rows: alone, alone, and with . The ratio of the joint row to the product of the individual gains is the interaction term. This ratio coming out below one is not a failure but information; by itself it tells the reader which composition is worth trying.
Second, write the regime next to the number. Batch, context length, which side of the KV crossover, and whether the target is latency or throughput belong on the same line as the factor. "" on its own is an incomplete sentence; " in this regime" is a complete one.
Third, declare the falsification threshold before measuring. Pre-registration is not only a tool of clinical trials. In my own work I wrote the threshold for the layout claim in advance: if the naive-layout row does not fall below the row where 4-bit quantization joins speculation, the layout contract comes off the contribution list. When the threshold is written before the measurement, the road to beautifying the result is closed; written after, the threshold is fitted to the result.
V. Closing
The consequence of this accounting in my own work was that the headline claim shrank. The twenty-four-fold product went. Sparsity moved out of the main claim into a conditional branch that depends on the overlap rate. What remains is the composition of quantization and speculation with their interference priced in. The number got smaller, but for the first time its domain of validity was stated.
The value of a model lies not in the speedup it promises but in the compositions it forbids. The accounting here forbids three things: adding sparsity inside a deep draft tree, carrying a factor measured in one regime into another, and multiplying factors whose interaction term has not been measured. These prohibitions are less showy than a list of gains, but they are what actually protects an engineer's time.
A companion essay to Software-Defined Bandwidth Expansion, where the composition law and the measurements behind these claims are worked out in full.