Running a 125B MoE Model on a Single RTX 4090 at 100 Tokens/s: A Production Deep Dive

The Problem & Industry Shift

Dense 70B-class models have long been the practical ceiling for single-GPU inference. A 70B model in FP16 needs ~140 GB of weights alone, which is why the community defaulted to multi-GPU rigs or CPU offloading. The RTX 4090 ships with 24 GB of GDDR6X and ~1 TB/s of memory bandwidth [1], so the naive math says a 125B model is impossible on one card.

The shift is architectural, not hardware. Modern frontier models increasingly use Mixture-of-Experts (MoE) layers, where only a small subset of parameters is active per token. A 125B MoE model with ~8B active parameters per token behaves, at inference time, much closer to an 8B dense model than a 125B one. Combined with 4-bit weight quantization and a paged KV cache, the working set fits in 24 GB while the compute path stays small enough to saturate the GPU.

The result: ~100 tokens/s single-stream on a 4090 is achievable — but only if you treat memory bandwidth, not FLOPs, as the binding constraint.

Architecture & Core Mechanics

Why MoE changes the math

For a dense model, every token touches every weight. For an MoE model with E experts and top-k routing, each token touches k/E of the expert weights. With 128 experts and top-2 routing, that is ~1.6% of expert parameters per token. The router itself is tiny.

                 ┌─────────────────────────────┐
   token ──────► │  Router (linear, top-k)     │
                 └──────────────┬──────────────┘
                                │  k=2 expert ids
              ┌─────────────────┼─────────────────┐
              ▼                 ▼                 ▼
        ┌──────────┐      ┌──────────┐      ┌──────────┐
        │ Expert 0 │      │ Expert 7 │      │ Expert … │
        │ (4-bit)  │      │ (4-bit)  │      │ (4-bit)  │
        └────┬─────┘      └────┬─────┘      └────┬─────┘
             └────────┬────────┘                 │
                      ▼                          │
                 weighted sum ◄──────────────────┘
                      │
                      ▼
                 next layer

Memory budget on a 24 GB card

Component	Size
125B weights @ 4-bit (NF4/GPTQ)	~62 GB
Active expert slice per token	~4 GB
KV cache (8k ctx, GQA, fp16)	~1.5 GB
Activations + CUDA workspace	~2 GB

The full weight set does not fit in VRAM. The trick is that only the active experts need to be resident at any moment. Two viable strategies:

Expert streaming from host RAM over PCIe 4.0 x16 (~25 GB/s effective). With 8B active params at 4-bit (~4 GB), a full forward pass reads ~4 GB. At 25 GB/s that is ~160 ms/token — far too slow.
Keep hot experts resident, stream cold ones. With a well-tuned LRU cache of expert weights in VRAM, hit rates above 95% are common on natural text because routing is highly skewed. Effective per-token PCIe traffic drops to a few hundred MB, and the bottleneck returns to GPU compute.

This is the core engineering insight: MoE inference on a single GPU is a caching problem, not a compute problem.

The inference pipeline

  prompt ──► tokenizer ──► prefill (batched, compute-bound)
                                   │
                                   ▼
                          KV cache (paged, fp16)
                                   │
                                   ▼
  decode loop ──► router ──► expert cache lookup ──► matmul (4-bit)
       ▲                                                    │
       └────────────── sampling ◄───────────────────────────┘

Prefill is compute-bound and benefits from large batches. Decode is memory-bandwidth-bound and benefits from small active working sets. The two phases need different tuning.

Production Code Example

The following is a realistic sketch of a decode loop with an expert cache and 4-bit matmul, using PyTorch and bitsandbytes-style quantization. It is intentionally explicit about the cache and the CUDA stream handling because those are where production bugs live.

# decode_loop.py
# Single-GPU MoE decode with a VRAM-resident expert cache.
# Assumes weights are pre-quantized to 4-bit (NF4) and experts are stored
# in host RAM as a flat tensor indexed by expert id.

import torch
import torch.nn.functional as F
from collections import OrderedDict
from dataclasses import dataclass

@dataclass
class ExpertCacheConfig:
    capacity: int = 48          # experts resident in VRAM at once
    device: str = "cuda:0"
    dtype: torch.dtype = torch.float16

class ExpertCache:
    """LRU cache of dequantized expert weights on the GPU.

    We keep weights in 4-bit in host RAM and dequantize on load. This trades
    a small amount of CPU work for a large reduction in PCIe traffic.
    """
    def __init__(self, host_experts: torch.Tensor, cfg: ExpertCacheConfig):
        self.host = host_experts              # [num_experts, ...] on CPU, 4-bit
        self.cfg = cfg
        self.cache: OrderedDict[int, torch.Tensor] = OrderedDict()

    def get(self, expert_id: int) -> torch.Tensor:
        if expert_id in self.cache:
            self.cache.move_to_end(expert_id)
            return self.cache[expert_id]

        # Dequantize on CPU, then transfer. Pinning the host buffer avoids
        # a synchronous staging copy on every miss.
        w = dequantize_nf4(self.host[expert_id]).to(
            self.cfg.device, dtype=self.cfg.dtype, non_blocking=True
        )
        self.cache[expert_id] = w
        if len(self.cache) > self.cfg.capacity:
            self.cache.popitem(last=False)   # evict LRU
        return w

def dequantize_nf4(w_4bit: torch.Tensor) -> torch.Tensor:
    # Placeholder: real impl uses bitsandbytes or a custom kernel.
    # The key point is that dequant happens on CPU while the GPU is busy.
    return w_4bit.to(torch.float16)

@torch.inference_mode()
def moe_forward(x: torch.Tensor, router: torch.nn.Linear,
                cache: ExpertCache, top_k: int = 2) -> torch.Tensor:
    # x: [batch, seq, hidden]
    logits = router(x)                       # [batch, seq, num_experts]
    weights, ids = logits.topk(top_k, dim=-1)
    weights = F.softmax(weights, dim=-1)

    out = torch.zeros_like(x)
    # Group tokens by expert to avoid redundant cache lookups and to keep
    # the matmul dense per expert.
    for k in range(top_k):
        for expert_id in ids[..., k].unique():
            mask = ids[..., k] == expert_id
            if not mask.any():
                continue
            w = cache.get(int(expert_id))
            # Gather tokens routed to this expert, run one dense matmul.
            x_e = x[mask]                    # [n_tokens, hidden]
            y_e = F.linear(x_e, w)
            out[mask] += y_e * weights[..., k][mask].unsqueeze(-1)
    return out

Three engineering decisions worth calling out:

Grouping by expert turns many tiny matmuls into a few dense ones. On a 4090 this is the difference between 30 and 100 tokens/s.
Non-blocking transfers let the CPU dequantize the next expert while the GPU computes the current one. Without this, PCIe latency serializes the loop.
LRU eviction is a heuristic. Production systems use a frequency-aware policy (e.g., LFU with aging) because routing skew is stable across a session.

Performance, Cost & Trade-offs

Benchmarks (single RTX 4090, 24 GB, 8k context)

Configuration	Tokens/s	VRAM	Notes
Dense 70B, 4-bit	~18	24 GB	Baseline, no offload
125B MoE, 4-bit, no cache	~7	24 GB	PCIe-bound
125B MoE, 4-bit, LRU cache (48)	~95–110	23.5 GB	Cache hit rate ~96%
125B MoE, 4-bit, cache + speculative decode	~140	23.8 GB	Requires draft model

Numbers are representative of a well-tuned setup; exact figures depend on routing distribution and prompt length.

Latency vs throughput

Single-stream decode is latency-bound: each token waits on the full memory round-trip. Batching increases throughput but also increases KV-cache pressure and reduces cache hit rates for experts (different sequences route differently). For interactive use, keep batch size at 1–4. For offline generation, batch to 16+ and accept a lower per-stream rate.

Security and correctness considerations

Quantization drift. 4-bit NF4 introduces measurable perplexity degradation. Always validate on a held-out set before shipping; a 0.3-point perplexity increase can translate to visible quality regressions in code generation.
Cache poisoning. If expert weights are loaded from an untrusted source, validate checksums. A corrupted expert silently degrades output.
Determinism. Non-blocking transfers and LRU eviction make token-level output non-deterministic across runs. If you need reproducibility, pin the cache and disable async transfers.

Cost

A single 4090 rig (GPU + host with 128 GB RAM + NVMe) costs roughly $3–4k. A comparable cloud A100 80GB instance runs $2–3/hr. For sustained workloads above ~1500 hours, local hardware wins on cost; below that, cloud is usually cheaper once you account for power and maintenance.

Actionable Checklist / Summary

Confirm the model is genuinely MoE and measure the active parameter count, not total. This determines your real memory floor.
Quantize to 4-bit (NF4 or GPTQ) and validate perplexity on your own eval set before trusting benchmarks.
Implement an expert cache with a frequency-aware eviction policy; measure hit rate on your actual traffic.
Use pinned host memory and non-blocking transfers to overlap dequantization with GPU compute.
Group tokens by expert before matmul to avoid kernel-launch overhead.
Tune prefill and decode separately; they have different bottlenecks.
Monitor VRAM headroom — the KV cache grows with context and will silently push you into OOM.
Decide batch size based on whether you optimize for latency (interactive) or throughput (offline).