---
title: "25x inference performance on NVIDIA GB300 NVL72"
url: https://stacklist.com/card/f02e1fa8-7830-4bc3-8d25-ea3034079f45
source_url: "https://www.linkedin.com/posts/paoloperrone_25x-inference-performance-on-nvidia-gb300-share-7478555102983827456-9R0I/?utm_source=share&utm_medium=member_ios&rcm=ACoAAAI21ZsBNnZPaKuTab7nquKLCveUW7o-1DE"
stack: https://stacklist.com/stack/4aae218c-38d7-4c05-b7ae-f2db0029a2e8
summary: "25x inference performance improvement on NVIDIA GB300 achieved through SGLang's RadixAttention mechanism, which optimizes KV cache reuse by indexing prefixes in a radix tree to eliminate redundant computations. For agent workloads with 80% prefix overlap, this approach combined with chunked prefill and continuous batching delivers significant throughput gains by computing shared system prompts and context only once instead of repeatedly."
tags: "inference-optimization, radix-attention, kv-cache, sglang, agent-workloads, serving-framework, gpu-performance"
key_entities: "Paolo Perrone (person), NVIDIA GB300 (technology), SGLang (technology), RadixAttention (technology), KV cache (concept), prefix-matching (concept), xAI (organization), NVIDIA (organization), AMD (organization), Microsoft Azure (organization), AWS (organization), Cursor (organization), agent-workloads (concept), Grok (technology)"
classification: "analysis"
content_hash: "sha256:98450f2f153065f6eb8e24d868e713ef6f53f6bff642f0cd97aec8fd0c4168e5"
acp_version: "0.2"
token_counts_approximate: 553
visibility: public
agent_accessible: true
status: "final"
---

# 25x inference performance on NVIDIA GB300 NVL72

Paolo Perrone 3d Report this post 25x inference performance on NVIDIA GB300 NVL72. Same hardware. Same model. Different serving framework. The framework: SGLang. The mechanism: RadixAttention. The expensive part of inference is the prefix. → System prompts → RAG context → Agent loop preambles Standard serving recomputes the KV cache for that prefix on every request. Computes the same key-value pairs. Throws them away. Does it again. RadixAttention keeps the KV cache indexed by prefix. New request comes in, the engine walks a radix tree, finds the longest token-prefix match, and starts generation from there. → Identical prefix: zero recomputation → Partial match: compute only the suffix For agent workloads, the math gets brutal. Every agent call shares the system prompt, tool definitions, memory. 4K-token system prompt. 100 concurrent agents. → RadixAttention computes the KV cache once → Standard serving computes it 100 times At 80% prefix overlap (typical for agent stacks), KV cache reuse alone delivers ~5x throughput. Add chunked prefill, paged attention, continuous batching → 25x on GB300. xAI runs Grok on it. So do NVIDIA, AMD, Cursor, Microsoft Azure, AWS. Inference cost isn't about model size anymore. It's about how well your serving stack exploits the structure of your workload. 💾 Save this before your next "why is my agent bill so high" debugging session. 38 5 Comments Like Comment Share Copy LinkedIn Facebook X Tayjus S 3d Report this comment the agent preamble is what turns prefix into a substrate. single-turn workloads never had it. Like Reply 1 Reaction Sairam Sundaresan 2d Report this comment As models mature, the competitive advantage is shifting toward how efficiently we serve them in production. Like Reply 1 Reaction Michał Piszczek 2d Report this comment 25x assumes your prefixes actually match. Multi-tenant traffic with different system prompts per customer gets a fraction of that reuse Like Reply 1 Reaction 2 Reactions Usman C. 2d Report this comment Paolo Perrone Bypassing the KV cache bottleneck for repetitive system prompts is where real scaling happens. Like Reply 1 Reaction See more comments To view or add a comment, sign in