Software-Defined Bandwidth Expansion: An Interference-Aware Effective-Bandwidth Calculus for Memory-Bound LLM Inference

Baturalp Güvenç · Wiener Labs · Istanbul, Türkiye · baturalp@wienerlabs.com

Abstract

Autoregressive large language model (LLM) inference at low batch size is memory-bandwidth bound: each generated token requires streaming the entire parameter set from high-bandwidth memory (HBM) to the compute cores, and the physical bandwidth $B_{\mathrm{phys}}$ is fixed by the silicon. We argue that the relevant quantity to optimize is not $B_{\mathrm{phys}}$ but the information carried per byte moved, and we formalize this as a software-defined effective bandwidth $B_{\mathrm{eff}}$ . We give an interference-aware composition law for $B_{\mathrm{eff}}$ that supersedes the naive product of per-mechanism speedups, and we show, both by a roofline argument and by a bit-exact kernel reference, that three software mechanisms (grouped low-bit weight streaming with on-chip dequantization, execution-guided block sparsity, and multi-token speculative reuse) compose sub-multiplicatively, and that block sparsity is net-negative unless it shares a memory layout co-designed with the dequantization path. Anchored on a training-free 4-bit post-training quantization (PTQ) configuration that runs on any existing checkpoint, the calculus predicts a wall-clock speedup with median $13\times$ (10-90 percentile $9.6$ - $17.1\times$ ) at batch one on a 70B model. It also predicts a negative result: at the optimal draft depth, execution-guided sparsity adds essentially nothing on top of speculation unless the sharing of skip decisions across drafted positions is measured to be high. All layout and correctness claims are verified in exact arithmetic on CPU; the GPU kernel skeleton maps line-for-line onto the verified reference.

Index Terms: LLM inference, memory bandwidth, quantization, dynamic sparsity, speculative decoding, GPU kernels, roofline model.

I. Introduction

The dominant cost of serving an autoregressive LLM at interactive latency, i.e. at batch size one, is not arithmetic but memory movement. To produce each token the decoder reads every weight of the model from HBM into on-chip SRAM exactly once; for a 70B model in FP16 this is $141$ GB per token, and on an accelerator delivering $B_{\mathrm{phys}} \approx 3.35$ TB/s the read alone floors the step time near $42$ ms regardless of how fast the tensor cores are. The arithmetic intensity of a matrix-vector product is $O(1)$ FLOP per byte, so the operating point sits deep in the memory-bound region of the roofline; the compute units are largely idle.

The physical remedy (wider buses, more stacks, faster HBM) is bounded by packaging and by economics, and improves slowly. This paper takes the software view: if the HBM $\to$ SRAM path cannot be physically widened, then the objective is to maximize the useful information transported per byte that crosses it. We name this target the software-defined effective bandwidth

B_{\mathrm{eff}} \;:=\; \frac{\text{bytes an FP16 dense baseline would move to achieve the observed token/s}}{\text{wall-clock time}} \qquad (1)

an operational, measurable quantity: $B_{\mathrm{eff}}$ is the bandwidth a hypothetical unoptimized engine would need in order to match the token/s that the optimized engine actually attains. A method that doubles tokens per second at fixed hardware doubles $B_{\mathrm{eff}}$ by definition, whether it did so by moving fewer bytes, by extracting more tokens per byte, or both.

A. The multiplicative fallacy

It is tempting to model the gain as a product of independent factors (compression $\times$ sparsity $\times$ speculation) and to report, for example, $4 \times 2 \times 3 = 24$ , hence $B_{\mathrm{eff}} = 24\,B_{\mathrm{phys}}$ . This is the central error we correct. The three mechanisms are not independent: they contend for the same bytes and the same memory-layout budget. A wider speculative tree enlarges the union of neurons that any token in the tree activates, which erodes the sparsity of the batched read; and skipping weights by a data-dependent predictor scatters the surviving accesses, which collapses coalescing and can leave the sparse path slower than the dense one. The true gain is therefore a constrained composition, not a product. Both couplings have been observed in isolation in prior work; what is missing, and what we supply, is the joint form in which all three mechanisms and the memory layout appear as one expression.

B. Contributions

An interference-aware $B_{\mathrm{eff}}$ calculus (Section II) that expresses the step time as a single roofline maximum over an effective byte count in which compression, sparsity, and speculation appear together with their mutual couplings. Each pairwise interaction has been observed separately in prior work; the contribution here is the closed-form three-way composition and the common currency $B_{\mathrm{eff}}$ in which the mechanisms can be compared and combined.
A layout co-design contract: we show, by a coalescing and bank-conflict simulation and by a bit-exact fused-kernel reference, that block-dynamic sparsity is net-positive only when packed low-bit weights and the sparsity mask share a neuron-contiguous, tile-transposed layout; the same simulation exhibits a configuration in which adding sparsity to a naive layout reduces throughput.
A PTQ-anchored, QAT-ceiling evaluation frame (Section III) that reports a reproducible, training-free operating point as the headline and a clearly labeled conditional ceiling for the training-based regime, so that the headline speedup claim is falsifiable and checkpoint-agnostic.

We emphasize scope. The regime of interest is low-batch, long-context, single-user or edge inference, where the read of weights and KV cache dominates. At large batch the operating point crosses into the compute-bound region and the compression gains amortize away; the calculus below makes that crossover explicit rather than hiding it.

II. The Effective-Bandwidth Model

A. Baseline step time

Let $P$ be the parameter count, $b$ the bits per stored weight, $\eta$ the achieved fraction of $B_{\mathrm{phys}}$ (memory efficiency), $F$ the peak compute rate, $C$ the context length in tokens, and $\kappa$ the bytes per token of KV cache. For dense FP16 greedy decoding the per-token step time is the roofline maximum

T_0 \;=\; \max\!\left( \frac{P \cdot \frac{16}{8} + C\kappa}{\eta\, B_{\mathrm{phys}}},\; \frac{2P}{F} \right), \qquad (2)

and at batch one the first argument dominates for all current accelerators.

B. The three mechanisms and their couplings

We introduce four software controls and, crucially, the terms that couple them.

(i) Low-bit streaming. Weights are stored at $b$ bits in groups of $G$ with an FP16 scale (and optional zero) per group, giving a true cost of $b + 16/G$ bits per weight; they are dequantized in registers after the load, so the HBM traffic scales as $b/16$ relative to FP16 but the arithmetic acquires a dequantization factor $d \gtrsim 1$ .

(ii) Execution-guided block sparsity. A lightweight router marks a fraction of the feed-forward blocks inactive for the current token; only the active blocks are fetched. Let $s$ be the per-token skip fraction and $\phi$ the share of parameters in the feed-forward sublayers.

(iii) Multi-token speculative reuse. A tree of $w$ candidate positions is drafted and verified against one read of the weights, yielding $\alpha$ accepted tokens per step; the effective work per step scales with $w$ while the read is amortized across $\alpha$ outputs.

The coupling. The sparsity seen by a batched, tree-shaped read is not $1-s$ but the complement of the union of active blocks over the tree. Writing $\rho$ for the fraction of the skip decision shared across sibling positions, the effective tree width for the union is $w_{\mathrm{eff}} = 1 + (w-1)(1-\rho)$ , and the retained-weight fraction is

\mathrm{keep}(s, w, \rho) \;=\; 1 - s^{\,w_{\mathrm{eff}}}, \qquad \psi \;=\; (1-\phi) + \phi \cdot \mathrm{keep}, \qquad (3)

where $\psi$ is the fraction of weight bytes actually streamed. Equation (3) is the quantitative form of the multiplicative fallacy: $\mathrm{keep} \to 1$ as $w$ grows, so speculation eats sparsity.

C. Composed step time and $B_{\mathrm{eff}}$

The optimized per-token step time is

T_{\mathrm{tgt}} \;=\; \max\!\left( \frac{P\frac{b}{8}\psi + C\kappa f + 2P_r}{\eta_s\, B_{\mathrm{phys}}},\; \frac{2P\psi w d + wC\kappa + 2P_r w}{F} \right), \qquad (4)

T \;=\; T_{\mathrm{tgt}} + D \cdot \max\!\left( \frac{P_d \frac{b_d}{8}}{\eta\, B_{\mathrm{phys}}},\; \frac{2P_d}{F} \right), \qquad B_{\mathrm{eff}} \;=\; \frac{\alpha\,(2P + C\kappa)}{T}, \qquad (5)

where $f \le 1$ is the KV compression factor, $\eta_s$ the memory efficiency under sparsity, $P_r$ the router parameters and $P_d$ the draft parameters. The second term of $T$ is the $D$ sequential draft passes, which are latency and do not amortize across the tree. The attention and router terms are small at short context and are not at long context. Equation (5) is exactly (1): FP16-dense bytes per token divided by wall-clock time per token.

Crucially $\alpha$ is not a free input. For per-token acceptance $p$ and draft depth $D$ the expected accepted tokens per verified step follow

\alpha(D) \;=\; \frac{1 - p^{D+1}}{1 - p},

so the model solves for the depth that maximizes $\alpha(D)/T(D)$ rather than assuming a tree width.

D. The layout contract: why $\eta_s$ is the crux

The single most consequential term in (4) is $\eta_s$ . Sparsity reduces $\psi$ but, if the surviving accesses are scattered, it also reduces $\eta_s$ , and the two effects can cancel. Concretely, a data-dependent skip that indexes weights in a neuron-interleaved layout forces a warp to gather one word per active neuron across a large stride; in a 128-byte segment model this drops attained bandwidth from $100\%$ to under $40\%$ (Section IV, and the simulation in the companion code). The break-even condition follows directly from (4): sparsity is net-positive only when

\eta_s \;\ge\; \eta_d \cdot \psi(s, w, \rho), \qquad (6)

i.e. the sparse layout must retain at least a $\psi$ -fraction of the dense efficiency. Since (3) makes $\psi \to 1$ at wide trees, at $w = 6$ , $s = 0.5$ , $\rho = 0.5$ one needs $\eta_s \gtrsim 0.93\,\eta_d$ : almost no coalescing may be lost. This is achievable only by co-design: packing the low-bit weights and the block mask into a single neuron-contiguous, tile-transposed layout so that the words a warp reads at a fixed column-group are unit-stride even after tiles are skipped. Under that layout the fused kernel attains full coalescing (verified bit-exactly against a dense reference), and (6) holds with margin at all tree widths. Absent it, the sparse path is slower than the dense one, and the $B_{\mathrm{eff}}$ of the "all mechanisms" configuration falls below that of compression-plus-speculation alone.

E. The context wall

As weights shrink under (i), the KV read $C\kappa f$ becomes the binding term in the memory numerator of (4). Setting weight bytes equal to KV bytes gives a crossover context

C^{\star} \;=\; \frac{P\,b}{8\kappa f};

for a 2-bit 70B model with FP16 KV this is on the order of $5 \times 10^4$ tokens, beyond which the KV cache dominates the read and further weight compression yields diminishing returns. KV compression ( $f < 1$ ) shifts $C^{\star}$ proportionally and is therefore a first-class mechanism, not an afterthought, in the long-context regime.

III. PTQ Anchor and QAT Ceiling

A. Why bit-width is an axis, not a constant

The compression factor $b$ trades throughput against model quality, and the trade is sharply nonlinear. Plain post-training quantization of an existing 70B checkpoint is essentially lossless at 4 bits (perplexity change below $0.2$ ), degrades noticeably at 3 bits, and is unusable at 2 bits (perplexity inflation of several points) unless one applies incoherence-processing or fine-tuning that is training-adjacent in cost. Weight formats that reach $1.58$ bits at full quality require pretraining from scratch and are not drop-in replacements for released weights. Reporting a single ambitious bit-width therefore conflates a reproducible engineering result with a research aspiration.

B. The evaluation frame

We treat $b$ as a measured axis and separate two operating points.

Anchor (reproducible, training-free). Grouped 4-bit PTQ, applicable to any released checkpoint with no retraining, combined with the co-designed layout of Section II, FP8 KV, and a speculative tree ( $w = 6$ , $\alpha \approx 2.6$ ) at a conservative skip $s \approx 0.35$ . This is the headline configuration; the calculus of (4) predicts a wall-clock speedup with median $\approx 13\times$ at batch one on a 70B model and a 10-90 percentile interval of $9.6$ - $17.1\times$ propagated over plausible ranges of acceptance, sharing and attained bandwidth. We report the interval rather than a point estimate.

Ceiling (projected, training-based). The same $B_{\mathrm{eff}}$ calculus evaluated at 2-bit quality-preserving quantization with a higher skip $s \approx 0.55$ projects toward $\approx 20\times$ ; we report it as a conditional upper envelope, explicitly labeled, to indicate how the method scales when a quantization-aware training budget is available. No claim of reproducibility is attached to this point.

C. A negative result: sparsity and speculation are near-exclusive

Solving (4) for the optimal depth exposes a consequence the multiplicative view hides. Standalone, execution-guided sparsity is worth about $+30\%$ at the anchor configuration. Its marginal value collapses as the tree deepens: $+16\%$ at $D = 1$ , $+8\%$ at $D = 2$ , $+1\%$ at $D = 4$ , and it turns negative beyond, because (3) drives $\psi \to 1$ while the router still costs bytes and arithmetic. Since the optimum sits at large $D$ , the two mechanisms are close to mutually exclusive: sparsity is not a multiplier on speculation but an alternative to it, and speculation wins at every acceptance rate we tested, down to $p = 0.4$ .

The exception is sharp and measurable. At high skip sharing the union grows slowly enough for sparsity to survive inside a shallow tree: at $D = 4$ its marginal value is $+8\%$ at $\rho = 0.8$ and $+21\%$ at $\rho = 0.95$ , against $-1.5\%$ at $\rho = 0.4$ . The question therefore reduces to the measured value of $\rho$ , which is why it is the first quantity in the calibration protocol and why the headline configuration is quantization plus speculation, with sparsity admitted only if $\rho$ is measured high or no draft model is available.

This framing keeps the headline falsifiable and checkpoint-agnostic while making the ceiling and its precondition (training) transparent. Table and kernel-level validation of the anchor point, and the calibration of $\alpha$ and $s$ on hardware, are the subject of the following sections.

IV. Kernel Design and the Layout Contract

This section specifies the artifact that makes $\eta_s$ large enough to satisfy (6), and reports the correctness and traffic properties we have verified for it.

A. Storage format

Weights are quantized in groups of $G$ columns with an FP16 scale and an optional FP16 zero point, and packed into 32-bit words, $\lfloor 32/b \rfloor$ weights per word. The true storage cost is $b + 16/G$ bits per weight for the symmetric variant and $b + 32/G$ for the asymmetric one; at $G = 128$ this is $4.125$ bits for the anchor configuration. Bit widths that do not divide 32 waste the residue: $b = 3$ costs $3.325$ effective bits rather than $3.125$ , which is why the design admits $b \in \{2, 4, 8\}$ only. The codec is idempotent on its own lattice, i.e. re-quantizing a dequantized tensor produces the same code words exactly, which we use as a correctness invariant.

B. Layout

Two constraints act in opposite directions. Dequantization wants the words a lane consumes to be adjacent in the $K$ direction; the router wants whole neurons to be addressable so that inactive ones are never fetched. We resolve them by storing weights neuron-contiguous and then tile-transposing: the packed array has logical shape $[\,n_{\mathrm{tiles}},\, WPR,\, B_N\,]$ , where $B_N$ is the number of output rows per program and $WPR = K / \lfloor 32/b \rfloor$ . A warp that reads the $B_N$ words belonging to a fixed column-group index therefore issues unit-stride loads, and a skipped tile is simply absent from the launch list.

This is the operative difference from a layout that indexes surviving neurons in the original row-major array. In a 128-byte segment model with a $40\%$ keep fraction at $b = 2$ , $K = 8192$ , the tile-transposed layout attains $100\%$ of the fetched segments useful, an unaligned variant attains $99.6\%$ , and a neuron-interleaved layout attains $39.9\%$ : the scattered variant wastes $2.5\times$ the bandwidth it saves. Substituting these into (6) reproduces the qualitative report in the literature that a naive sparse path in the original weight layout is slower than a layout-aware one, and gives the quantitative threshold at which the sign flips.

On the SRAM side, a $32 \times 32$ FP32 tile loaded by column induces a 32-way bank conflict at stride 32; padding the row stride to 33, or applying an XOR swizzle to the column index, reduces the worst-case conflict degree to 1. Both are compatible with the tile-transposed global layout, since they act only on the staging buffer.

C. Fused kernel

One program instance owns one active tile and performs the whole chain in a single pass over HBM: load packed words, unpack $\lfloor 32/b \rfloor$ lanes in registers, apply the group scale and zero, and accumulate the product with the activation vector. Dequantized weights never touch memory. Execution-guided sparsity is realized as a smaller launch grid rather than a masked load, so an inactive tile generates no weight traffic at all, which is the strongest available form of the mechanism and the one assumed by $\psi$ in (4).

We validated the kernel logic against a dense dequantize-then-multiply reference on identical inputs, obtaining agreement to $1.9 \times 10^{-6}$ relative error, i.e. exact up to FP16 rounding, and confirmed that the number of words touched equals $n_{\mathrm{active}} \cdot WPR \cdot B_N$ exactly. In a representative configuration ( $N = 1024$ , $K = 4096$ , $b = 2$ , $B_N = 64$ , 6 of 16 tiles active) the kernel moves $18.96\times$ fewer bytes than an FP16 dense pass, decomposing into $21.3\times$ from packing and $2.5\times$ from tile skipping, less the scale and zero overhead. The GPU port maps line for line onto this reference, so the port carries the same correctness argument and the same traffic ledger.

D. What is not yet optimized

The reference establishes correctness, traffic, and coalescing, not peak throughput. Two known inefficiencies remain: the per-lane unpacking loop is scalar and must be vectorized onto tensor-core fragments, and group scales are currently reloaded per lane rather than once per word. Both are local transformations that do not affect the layout contract, and both are scheduled for the hardware phase, where $\eta_s$ is measured directly rather than inferred.

V. Related Work and Position

Each mechanism in Section II, and each pairwise interaction between them, has prior art; we state this explicitly because it determines what remains to be shown.

Low-bit streaming. Weight-only PTQ at 4 bits is mature and essentially lossless, and dedicated mixed-precision GEMM kernels realize the bandwidth saving. Quality-preserving 2-bit PTQ requires incoherence processing or a fine-tuning-adjacent stage, and sub-2-bit formats at full quality require pretraining from scratch; this is the basis of the frame in Section III.

Execution-guided sparsity. Predictor-driven skipping of feed-forward neurons is established, as are its GPU-side difficulties: fine-grained branching and scattered access dominate unless the layout is restructured. Systems that combine grouped quantization with dynamic activation sparsity report that a naive sparse path in the original weight layout is markedly slower than a layout-aware one, which is the empirical counterpart of the break-even condition (6).

Union growth. That the union of active neurons grows when several positions are served from one weight read is known from the batched setting, where union sparsity is observed to diminish as batch size increases. Conversely, work on activation sparsity reports substantial reuse of already-activated neurons across speculatively drafted positions, i.e. a high value of $\rho$ in (3); the sign of this effect is therefore favourable and must be measured, not assumed.

Speculation meets quantization. It has been reported that the extra arithmetic of tree-shaped draft verification erodes the memory-side benefit of 4-bit quantization, limiting the compatibility of the two techniques. Equation (4) is the closed form of exactly that observation: the compute argument scales with $w\,d$ while the memory argument scales with $b\,\psi$ , so the roofline maximum switches arguments at a tree width that the calculus predicts.

Roofline systematizations. Roofline analyses of LLM decoding, of sparsity-accuracy trade-offs, and amortization models for speculative decoding with offloading all treat one or two levers at a time.

Position. Against this background we do not claim any individual mechanism, nor the existence of pairwise interference. What is unclaimed, to our knowledge, is the single three-way composition of low-bit streaming, execution-guided sparsity and speculative reuse in one calculus, together with the explicit layout break-even condition (6) and the use of $B_{\mathrm{eff}}$ as a common currency across mechanisms. A first consistency check is encouraging: at $w = 1$ , $s = 0.5$ and $\phi \approx 0.70$ the calculus predicts a $1.54\times$ sparsity gain over a dense 4-bit baseline, against published measurements clustering near $1.44\times$ , implying $\eta_s/\eta_d \approx 0.94$ once router overhead is absorbed, comfortably above the break-even value $\psi = 0.65$ .

Reproducibility

The composition model (2)-(4), the layout coalescing and bank-conflict simulations, the grouped-quantization codec, and the fused dequantize-sparsify-multiply kernel (checked bit-for-bit against a dense reference, with the GPU port mapping line-for-line onto the verified CPU code) are provided as a companion code repository in exact, dependency-light form.