Stay updated on AI infrastructure, inference techniques, and performance optimization.

<p>Subscribe to our <u>newsletter</u></p>

Expert how-tos, deep-dive guides, and real-world stories from the Bento team, to help you build and scale AI at blazing speed.

Chaoyu Yang

For inference, the DIY approach rarely delivers the control it promises. Instead, it drains time, budget, and talent, often becoming the very bottleneck it was meant to solve.

Bento x Build or Buy.png

Should You Build or Buy Your Inference Platform?

Sherlock Xu

Explore the best open-source LLMs and find answers to common FAQs about performance, inference optimization, and self-hosted deployment.

oss-llms.png

Top-Rated LLMs for Chat in 2025

<p>NVIDIA has long been the default choice for GPUs in AI. Still, AMD is working to carve out space in high-performance computing (HPC) and enterprise AI. For teams exploring alternatives to NVIDIA, the AMD MI-series GPU accelerators offer strong hardware capabilities and impressive memory bandwidth.</p><p>The main challenge lies in the AMD ecosystem. The NVIDIA CUDA platform remains the industry standard, while AMD’s ROCm software stack is still maturing. That said, with the launch of the MI300X and other CDNA-based GPUs, AMD has become a serious contender worth evaluating.</p><p>In this post, we’ll break down the GPU lineups of AMD, focusing on data center GPUs, and then compare models from the MI100 to the MI300X for different AI use cases.</p><h2>AMD GPU lineup</h2><p>Similar to NVIDIA, AMD organizes GPUs into three main categories:</p><h3>Radeon (Consumer/Gaming)</h3><p>Radeon GPUs are designed for gaming, graphics, and general-purpose computing. Some developers also experiment with them for AI models, but they’re not suitable for large-scale inference or enterprise workloads. Recent models include the Radeon RX 9070 XT, RX 9070, and RX 9060 XT.</p><h3>Radeon Pro (Professional)</h3><p>Radeon Pro GPUs target professional creators, engineers, developers and AI professionals. They power workloads like 3D rendering, video editing, and CAD. While they are not typically deployed in big data centers, they can be used for small-scale AI inference workloads or workstation environments. Current products include the Radeon Pro R9000 series (32 GB of memory; RDNA 4) and W7000 series (up to 48 GB of memory; RDNA 3).</p><h3>Instinct MI-Series (AMD Data Center GPUs)</h3><p>The Instinct MI-series is AMD’s line of GPUs built for enterprise-scale AI, HPC, and cloud computing. AMD builds its data center GPUs on a dedicated architecture called CDNA (Compute DNA). Unlike RDNA, which powers gaming and workstation cards, CDNA targets HPC and AI workloads. The focus is on raw compute performance, memory bandwidth, and scaling across multiple accelerators rather than graphics rendering.</p><p>Notable products include the MI210, MI250, MI300X and MI325X. The latest MI350 series (MI350X, MI355X), launched in June 2025, features 288 GB of HBM3E memory and offers day-zero support for major AI frameworks, libraries, and cutting-edge models.</p><p>In short: Radeon is for gaming, Radeon Pro is for professional creators, and the Instinct MI-series is AMD’s answer to <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/nvidia-data-center-gpus-explained-a100-h200-b200-and-beyond">NVIDIA data center GPUs</a>.</p><h2>What is AMD ROCm?</h2><p>One of the biggest differences between AMD and NVIDIA isn’t just the hardware; it’s the software ecosystem. NVIDIA has CUDA, which has become the industry standard for GPU programming. AMD’s answer is ROCm (Radeon Open Compute Platform).</p><p>ROCm is an open-source software stack that supports the CDNA architecture. It provides drivers, libraries, and tools that enable developers to run HPC and AI workloads on AMD GPUs. With ROCm, frameworks like PyTorch and TensorFlow can use AMD hardware. This is similar to how they work with CUDA.</p><p>While ROCm has matured significantly, it still trails CUDA in ecosystem size and polish. As mentioned previously, this gap has been one of the biggest challenges for AMD in competing with NVIDIA.</p><h2>AMD data center GPU naming conventions</h2><p>The naming system for AMD data center GPUs is more straightforward than NVIDIA’s.</p><ul><li><strong>Prefix</strong>: Instead of letters tied to architectures (like A100 for Ampere), AMD uses the MI prefix, short for Machine Intelligence.</li><li><p><strong>Numbers</strong>: The number following “MI” indicates the generation:</p><ul><li>MI100 → 1st generation (CDNA)</li><li>MI200 → 2nd generation (CDNA 2)</li><li>MI300 → 3rd generation (CDNA 3)</li><li>MI350 → 4th generation (CDNA 4)</li></ul><p>Within each generation, higher model numbers (like MI250 vs. MI210) generally mean more powerful hardware, additional memory, or a different performance tier.</p></li><li><strong>Suffix</strong>: AMD often adds suffixes like X (e.g., MI250X, MI300X, MI350X) to indicate higher-performance versions with more memory or computing power than the base model.</li></ul><h2>AMD GPU comparison: MI250X vs. MI300X vs. MI325X vs. MI350X</h2><p>Here are some of the most popular AMD data center GPUs for AI inference:</p><figure class="table"><table style="border-collapse:collapse;border-color:hsl(0, 0%, 0%);border-width:1.5px;text-align:left;"><thead><tr><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:12%;"><strong>GPU</strong></th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:15%;"><strong>Architecture</strong></th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:18%;"><strong>Memory</strong></th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:20%;"><strong>Peak Memory Bandwidth</strong></th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:35%;"><strong>Best For</strong></th></tr></thead><tbody><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">MI100</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">CDNA</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">32 GB HBM2</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">1.2 TB/s</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Entry-level HPC</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">MI250X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">CDNA 2</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">128 GB HBM2e</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">3.2 TB/s</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Mid-range AI inference</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">MI300X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">CDNA 3</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">192 GB HBM3</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">5.3 TB/s</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">High-performance LLM training &amp; inference, HPC</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">MI300A</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">CDNA 3</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">128 GB HBM3 (unified)</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">5.3 TB/s</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Integrated AI/HPC workloads</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">MI325X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">CDNA 3</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">256 GB HBM3E</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">6 TB/s</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Advanced LLM training &amp; inference, FP8</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">MI350X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">CDNA 4</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">288 GB HBM3E</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">8 TB/s</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Ultra-large models, long-context inference, next-gen AI</td></tr></tbody></table></figure><p>&nbsp;</p><p>In practice, availability and software support (via ROCm) will play a big role in determining which GPU is the best fit for your team.</p><h2>How much do AMD MI-Series GPUs cost?</h2><p>You can find AMD GPUs on several GPU cloud providers and public cloud platforms. However, not all providers carry the full range of AMD data center GPUs, and pricing can differ significantly depending on region, provider, and commitment length.</p><p>Here is the latest pricing information for the AMD MI325X and MI300X GPUs:</p><figure class="table"><table style="border-collapse:collapse;border-color:hsl(0, 0%, 0%);border-width:1.5px;text-align:left;"><thead><tr><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:20%;">Provider</th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:25%;">GPU</th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:30%;">Billing Model</th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:25%;">Price</th></tr></thead><tbody><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Vultr</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">AMD MI325X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">On-demand</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$4.615/GPU/hr</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Vultr</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">AMD MI325X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">36-month prepaid</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$2.00/GPU/hr</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Vultr</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">AMD MI300X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">On-demand</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$3.99/GPU/hr</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Vultr</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">AMD MI300X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">24-month prepaid</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$1.85/GPU/hr</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Oracle Cloud</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">AMD MI300X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">On-demand</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$6.00/GPU/hr</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">DigitalOcean</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">AMD MI300X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">On-demand</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$1.99/GPU/hr</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">DigitalOcean</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">AMD MI300X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">12-month commitment</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$1.49/GPU/hr</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">DigitalOcean</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">AMD MI325X</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">12-month commitment</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$1.69/GPU/hr</td></tr></tbody></table></figure><p>&nbsp;</p><blockquote><p>Note: Pricing data was collected from <a target="_blank" rel="noopener noreferrer" href="https://www.vultr.com/pricing/">Vultr</a>, <a target="_blank" rel="noopener noreferrer" href="https://www.oracle.com/cloud/price-list/">Oracle Cloud</a>, and <a target="_blank" rel="noopener noreferrer" href="https://www.digitalocean.com/pricing/gpu-droplets">Digital Ocean</a> on September 4, 2025, and may change over time.</p></blockquote><p>Unlike NVIDIA T4, A100, or H100 GPUs, standardized on-demand pricing for AMD GPUs is harder to find. Still, the MI300X and MI350 series are widely seen as direct competitors to NVIDIA H100 and H200. They often offer a lower cost per unit of memory bandwidth. For enterprise buyers, the total cost of ownership (TCO) may be more favorable with AMD, especially for long-term workloads that benefit from large HBM3/HBM3E memory pools.</p><p>&nbsp;</p><hr><p>&nbsp;</p><p>Now let’s take a look at some of the most common questions about AMD data center GPUs.</p><h2>What is the best AMD GPU?</h2><p>There’s no single “best” AMD GPU. It depends on your workload. For smaller-scale HPC or AI inference, the MI210 or MI250 may be sufficient. For large-scale training and long-context LLM inference, the MI300X (192 GB HBM3) and the MI350X (288 GB HBM3E) are AMD’s top-end options. The best AMD GPU is the one that balances performance, memory, and cost for your specific use case.</p><h2>Is AMD better than NVIDIA?</h2><p>Whether AMD or NVIDIA is “better” depends on your priorities:</p><ul><li>NVIDIA GPUs (like the A100, H100, and H200) dominate in software support, thanks to the CUDA ecosystem and deep framework integration.</li><li>AMD GPUs (MI300X, MI350X) often provide more memory at a lower cost per GPU. This makes them attractive for memory-bound AI workloads such as LLMs with long context windows.</li></ul><p>In practice, NVIDIA is still the most widely adopted in AI, but AMD is increasingly competitive in HPC and enterprise AI inference.</p><h2>MI300X vs. H100</h2><p>The AMD Instinct MI300X and NVIDIA H100 are often compared because they target similar high-performance AI workloads:</p><ul><li>Memory: MI300X offers 192 GB HBM3, while H100 provides 80 GB HBM2e (with the H200 extending this to 141 GB). This makes MI300X particularly strong for long-context LLMs.</li><li>Ecosystem: H100 has the advantage of CUDA and broad adoption across cloud providers. MI300X relies on ROCm, which has matured but is still catching up in ecosystem support.</li><li>Performance: Benchmarks vary by workload. However, the MI300X is a strong competitor, especially for memory-heavy AI tasks.</li></ul><p>Before choosing a GPU for AI inference in production, it’s essential to benchmark against your own workloads and goals. This is especially important for optimization with advanced distributed inference techniques such as <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/llm/inference-optimization/prefix-caching">prefix caching</a>, <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/llm/inference-optimization/kv-cache-offloading">KV cache offloading</a>, or <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/llm/inference-optimization/prefill-decode-disaggregation">prefill–decode disaggregation</a>.</p><h2>Conclusion</h2><p>AMD is no longer just a challenger in the GPU market. It’s becoming an essential part of the conversation for AI infrastructure. For teams limited by the supply or pricing of NVIDIA GPUs, the AMD MI300X and MI350 series provide a credible and cost-effective alternative.</p><p>The real question isn’t “AMD or NVIDIA?” Instead, it is how to build an inference stack that can flexibly adapt to changing models, frameworks, and optimization techniques. Whether or not this inference layer can help you smoothly run workloads with these new technologies may be the most strategic choice of all.</p><p>At <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/">Bento</a>, our unified inference platform helps enterprises do exactly that. AI teams can seamlessly tap into the most suitable NVIDIA or AMD GPUs at the best rates and availability. Also, they can apply custom inference optimizations to get optimal performance with speed and ease for their unique use cases.</p><p>Learn more:</p><ul><li>Read our <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/llm/">LLM Inference Handbook</a> for practical optimization strategies</li><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/join-slack-from-blog-amd-data-center-gpus">Join our Slack community</a> to connect with other AI practitioners</li><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/contact-us-from-blog-amd-data-center-gpus">Schedule a call with our experts</a> if you need tailored advice on GPU inference</li></ul>

Explore AMD GPUs for AI inference. Learn about MI250X, MI300X, MI350X, pricing, performance, and how they compare to NVIDIA for AI and HPC.

amd-gpu-cover-image.png

AMD Data Center GPUs Explained: MI250X, MI300X, MI350X and Beyond

<p>For AI teams that want to self-host Generative AI (GenAI) models like LLMs, one of the most important choices is which GPU to use.</p><p>In the GPU industry, NVIDIA has established itself as the undisputed leader, especially for AI workloads. It offers a wide range of GPUs that can run AI workloads of different sizes. With so many options, it can be difficult to compare performance and cost, and making the wrong choice can be costly.</p><p>This is particularly true for workloads like <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/inference-platform-the-missing-layer-in-on-prem-llm-deployments">on-prem LLM deployments</a>. Once you invest in the hardware, it’s hard to undo the decision. Nobody wants to sink money into GPUs that don’t deliver the expected value.</p><p>In this post, we’ll break down NVIDIA’s GPU product lines, focusing on data center GPUs for AI inference. We’ll compare the most common ones and explain how to weigh performance, memory, and cost for your use case.</p><h2>What are NVIDIA data center GPUs</h2><p>When people talk about GPUs for AI inference, they usually refer to NVIDIA data center GPUs. These cards are the backbone of modern AI infrastructure and you’ll often find them in cloud servers and enterprise data centers. They are optimized for large-scale AI workloads and can be shared across teams or scaled to massive clusters.</p><p>NVIDIA updates its data center lineup every one to three years. Each new generation improves memory, throughput, and efficiency. Well-known examples include the T4, L4, A100, H100, H200, and the new B200. These GPUs dominate AI inference benchmarks and show up on almost every GPU comparison chart for LLMs and GenAI use cases.</p><p>Many people confuse data center GPUs with other NVIDIA products:</p><ul><li><strong>GeForce (Consumer/Gaming)</strong>. Graphic cards like the RTX 5090 or 4090 are built for gaming and general use. They perform well, and some users do run GenAI workloads on them. However, they aren’t made for enterprise-scale AI deployment.</li><li><strong>RTX (Professional).</strong> Formerly branded as Quadro, these GPUs target creative professionals in fields like visualization, architecture, and 3D design. They’re built for stability and certified software support, but they’re not aimed at large-scale AI training or inference.</li></ul><p>In short, GeForce is for gamers, RTX is for professional creators, and data center GPUs are for AI and large-scale computing. If your goal is to train or serve GenAI models, data center GPUs are the right choice.</p><h2>GPU architectures: Turing, Ampere, Hopper, Lovelace, Blackwell</h2><p>Each time NVIDIA launches a new GPU generation, it brings a new <a target="_blank" rel="noopener noreferrer" href="https://www.nvidia.com/en-us/technologies/">architecture</a> that improves performance, efficiency, and support for AI workloads. The latest Blackwell architecture packs 208 billion transistors, built on TSMC’s custom 4NP process. Blackwell GPUs feature two reticle-limited dies connected by a 10 TB/s chip-to-chip interconnect, allowing them to operate as a unified single GPU.</p><p>A quick way to identify NVIDIA data center GPUs is through their names. The naming convention follows a simple pattern that reveals key information about the chip:</p><ul><li><strong>First letter</strong>: Indicates the architecture generation. NVIDIA names each generation after a well-known scientist or mathematician, like Ada Lovelace and Grace Hopper.<ul><li>Turing (T4)</li><li>Ampere (A10, A100)</li><li>Hopper (H100, H200)</li><li>Lovelace (L4)</li><li>Blackwell (B200)</li></ul></li><li><strong>Numbers</strong>: The number following the letter usually indicates the model’s relative position in the lineup. Within the same generation, higher numbers generally mean more powerful or feature-rich GPUs.</li><li><strong>Memory suffix</strong>: Some models include memory capacity directly in the name, such as the A100 80GB, which helps distinguish variants.</li></ul><h2>Best NVIDIA GPUs for AI inference: A100, H100, H200, B200 compared</h2><p>Not all NVIDIA GPUs are created for the same purpose. Within each architecture, NVIDIA offers several models aimed at different price, performance, and power targets. In general, higher-numbered models provide more compute power, memory, and features, but they also come at a higher cost.</p><p>Here are some of the most common NVIDIA data center GPUs used for AI today:</p><figure class="table"><table style="border-collapse:collapse;border-color:hsl(0, 0%, 0%);border-width:1.5px;text-align:left;"><thead><tr><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:15%;">GPU</th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:20%;">Architecture</th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:15%;">Memory</th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:50%;">Best For</th></tr></thead><tbody><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;"><strong>T4</strong></td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Turing</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">16 GB</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Entry-level inference</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;"><strong>L4</strong></td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Lovelace</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">24 GB</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Energy-efficient inference</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;"><strong>A10</strong></td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Ampere</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">24 GB</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Mid-range inference, AI training</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;"><strong>A100</strong></td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Ampere</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">40 &amp; 80 GB</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">High-performance LLM training &amp; inference, HPC</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;"><strong>H100</strong></td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Hopper</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">80 GB</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Advanced LLM training &amp; inference, FP8</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;"><strong>H200</strong></td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Hopper</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">141 GB</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Ultra-large models, long-context inference</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;"><strong>B100</strong></td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Blackwell</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">192 GB</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Next-gen AI training, inference, HPC</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;"><strong>B200</strong></td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Blackwell</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">192 GB</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">Frontier-scale AI, multi-trillion parameter models</td></tr></tbody></table></figure><h3>GPU benchmarks and comparisons</h3><p>When choosing between GPUs, benchmark data is one of the most useful points of reference. For LLM inference, the most common metrics are throughput (e.g., Tokens Per Second) and latency (e.g., Time to First Token). NVIDIA publishes general performance numbers for each generation, but they don’t always reflect the nuances of real-world GenAI workloads.</p><p>That’s why independent GPU comparisons can be especially useful. Benchmarking platforms and inference frameworks (e.g., vLLM and SGLang) often measure GPUs against specific LLMs or inference scenarios. These results can help you see how different compute accelerators perform under practical conditions.</p><h3>GPU price considerations (T4 vs. A100 vs. H100)</h3><p>Performance is only part of the equation, and pricing is another key factor. GPU costs can vary widely depending on the provider and purchase model. Many vendors offer on-demand, 1-year, and 3-year committed plans, with large discounts for longer commitments.</p><p>For example, here’s how the cost of a T4 vs. A100 vs. H100 compares on Google Cloud (us-central1):</p><figure class="table"><table style="border-collapse:collapse;border-color:hsl(0, 0%, 0%);border-width:1.5px;text-align:left;"><thead><tr><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:35%;">GPU</th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:20%;">On-Demand</th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:22.5%;">1-Year Commitment</th><th style="background-color:hsl(0, 0%, 90%);border-bottom-style:solid;border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;width:22.5%;">3-Year Commitment</th></tr></thead><tbody><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">NVIDIA T4 (16 GB)</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$255.50/mo</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$160.60/mo</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$116.80/mo</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">NVIDIA A100 (40 GB) — 1× A100 in <code>a2-highgpu-1g</code> VM</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$2,681.57/mo</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$1,689.37/mo</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$938.57/mo</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">NVIDIA A100 (80 GB) — 1× A100 in <code>a2-ultragpu-1g</code> VM</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$3,700.22/mo</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">N/A</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">N/A</td></tr><tr><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">NVIDIA H100 (80 GB) — 8× H100s in <code>a3-highgpu-8g</code> VM</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$64,597.70/mo (≈$8,074.71 per GPU)</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$44,810.08/mo (≈$5,601.26 per GPU)</td><td style="border-color:hsl(0, 0%, 0%);border-width:1.5px;padding:8px;">$28,371.00/mo (≈$3,546.38 per GPU)</td></tr></tbody></table></figure><p>&nbsp;</p><blockquote><p>Note: Pricing was collected on August 27, 2025, and may change. For the latest details, check <a target="_blank" rel="noopener noreferrer" href="https://cloud.google.com/products/calculator?hl=en">Google Cloud's pricing calculator</a>.</p></blockquote><p>These numbers make the contrast clear. An entry-level <strong>NVIDIA T4 GPU</strong> costs just a few hundred dollars per month; in contrast, a high-end <strong>H100 cluster</strong> can exceed <strong>$60,000/month</strong>. Ultimately, choosing the right GPU is about balancing performance requirements and cost efficiency, especially for long-term AI deployments.</p><h3>Why GPU memory matters</h3><p>When running LLM inference, GPU memory (VRAM) often matters more than raw compute power. One of the biggest reasons is the <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/llm-inference-basics/how-does-llm-inference-work">KV cache</a>. It stores attention keys and values for every token in the input sequence. This way, the model doesn’t need to recalculate them at each decoding step. The result is faster inference, but at the cost of high memory usage.</p><p>The KV cache size increases linearly with sequence length. This means that long-context scenarios can quickly use up available GPU memory, turning VRAM into the main bottleneck. The amount of GPU memory you have directly determines the size of the models and context windows you can support. For example:</p><ul><li>H200 (141 GB): Large enough to handle extended context windows without offloading the KV cache to external memory systems.</li><li>T4 and A10 (16–24 GB): Ideal for smaller inference jobs, but they can struggle when sequences get longer.</li></ul><p>Because of these constraints, the AI community is actively exploring ways to get around memory limits, such as <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/prefill-decode-disaggregation">prefill-decode disaggregation</a>, <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/kv-cache-offloading">KV cache offloading</a>, and <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/pagedattention">memory-efficient attention</a> techniques.</p><p>In short, if your workloads involve long prompts or large batch inference, GPU memory capacity is often just as important as raw compute power when choosing a graphic card.</p><h2>AMD GPU alternatives</h2><p>While NVIDIA currently dominates the AI GPU market, it’s not the only option. AMD’s MI-series accelerators have made steady progress and now offer competitive performance for certain tasks, particularly in AI training and inference. The challenge for AMD is less about hardware capability and more about software support. The NVIDIA CUDA ecosystem has become the default standard for AI development, with broad framework integration and community adoption.</p><p>We covered the MI-series GPUs in more detail <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/amd-data-center-gpus-mi250x-mi300x-mi350x-and-beyond">in a separate blog post</a>.</p><hr><p>&nbsp;</p><p>Now let’s take a quick look at some of the common questions about NVIDIA data center GPUs.</p><h2>What is the best NVIDIA GPU for data centers?</h2><p>The “best” GPU depends on your workload. For entry-level inference, the T4 or L4 may be sufficient. For fast, high-throughput inference, the A100 and H100 remain industry standards. The H200 and B200 are more advanced and ideal for long-context LLMs and frontier-scale AI models.</p><h2>What are the GPU data center companies?</h2><p>NVIDIA supplies data center GPUs to all major cloud providers (AWS, Google Cloud, Microsoft Azure) and to enterprise customers building their own clusters. Other companies like AMD (MI-series GPUs) and Intel (Gaudi accelerators) also offer data center GPUs or accelerators. However, NVIDIA currently dominates the market with the broadest ecosystem and adoption.</p><h2>Should I build my own GPU data center for AI inference?</h2><p>Building your own GPU data center can give you more control and potentially lower costs at scale. However, it also means complexity in deployment, scaling, and operations. If you’re considering this path, check out these deep dives:</p><ul><li><a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/how-to-beat-the-gpu-cap-theorem-in-ai-inference">How to Beat the GPU CAP Theorem in AI Inference</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/inference-platform-the-missing-layer-in-on-prem-llm-deployments">On-Prem LLM Deployments</a></li></ul><h2>Conclusion</h2><p>NVIDIA data center GPUs cover a wide spectrum, from entry-level inference on the T4 to frontier-scale inference on the B200. We hope this post gives you a clearer view of the product line and the key considerations when choosing GPUs.</p><p>At Bento, we offer a wide range of GPU options across regions and cloud providers for AI inference. Our inference platform automatically matches your workloads with the best available GPUs at the most competitive rates, and scales resources up or down as your traffic changes.</p><p>Learn more:</p><ul><li>Read our <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/">LLM Inference Handbook</a> to learn practical strategies for faster, cheaper inference.</li><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/join-slack-from-blog-nvidia-data-center-gpus">Join our Slack community</a> and connect with other AI teams working on GPU inference.</li><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/contact-us-from-blog-nvidia-data-center-gpus">Schedule a call with our experts</a> to discuss the right GPU strategy for your workloads.</li></ul>

Understand NVIDIA data center GPUs for AI inference. Compare T4, L4, A100, H100, H200, and B200 on use cases, memory, and pricing to choose the right GPU.

nvidia-data-center-gpus.jpg

NVIDIA Data Center GPUs Explained: From A100 to B200 and Beyond

Understand the differences between DeepSeek-V3, R1, V3.1, and distilled models. Learn how to choose the right model and deploy them securely with BentoML.

deepseek-models.png

The Complete Guide to DeepSeek Models: V3, R1, V3.1 and Beyond

Aaron Pham

Frost Ming

Larme Zhao

<p>LLMs are powerful, but they’re slow during inference. That’s because they’re trained with an autoregressive pattern: generating the next token based on all the previous ones. During inference, this means every new token requires a full forward pass through the model, followed by sampling and then appending that token to the input for the next step. This process is inherently sequential, meaning the model can’t compute future tokens ahead even if the GPU is idle. This leads to high <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/llm-inference-metrics">Inter-Token Latency (ITL)</a> and poor GPU utilization.</p><p>Speculative decoding offers a solution. By having a small draft model predict several tokens in advance, and letting a larger target model verify them in parallel, you can accelerate the token generation process.</p><p>In practice, however, we found speculative decoding only delivers the expected speedup if the draft model’s distribution matches closely with the target model. The key is using the right draft model for your workload, which in many real-world cases means training one on your own data.</p><p>In this post, we’ll walk through:</p><ul><li>How speculative decoding works and when it helps</li><li>Why acceptance rate is the performance bottleneck</li><li>Our benchmarks with vLLM, SGLang, and EAGLE 3</li><li>Why training a custom draft model is helpful for real-world speedups</li></ul><h2>What is speculative decoding</h2><p>Speculative decoding is an inference-time optimization technique that speeds up LLM token generation without sacrificing output quality. It’s inspired by the concept of speculative execution, where operations are computed in parallel ahead of time and discarded if unneeded.</p><p>This technique builds on <a target="_blank" rel="noopener noreferrer" href="https://research.google/blog/looking-back-at-speculative-decoding/">two key observations about LLM inference</a>:</p><ol><li><strong>LLM inference is memory-bound</strong>. Hardware like GPUs can perform hundreds of trillions of operations per second, but their memory bandwidth is much lower. That means these chips often have unused compute capacity while waiting on memory. For transformer-based LLMs, they can only perform a few operations per byte of memory read during inference, which means compute is underutilized.</li><li><strong>Some tokens are easier to predict than others.</strong> Many next tokens are obvious from context and can be easily handled by a small model.</li></ol><p>To take advantage of these facts, speculative decoding uses a draft-then-verify paradigm:</p><ul><li><strong>Draft model:</strong>&nbsp;A smaller, faster model proposes a draft sequence of tokens.</li><li><strong>Target model:</strong>&nbsp;The original larger model verifies the draft’s tokens at once and decides which to accept.</li></ul><blockquote><p>Note: This draft-then-verify idea was first introduced by <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/abs/1811.03115">Stern et al. (2018)</a>, and later extended into a statistically grounded technique called speculative sampling by <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/pdf/2302.01318">DeepMind</a>. Speculative decoding is the application of speculative sampling to inference from autoregressive models, like transformers.</p></blockquote><p>Here is how it works in more details:</p><figure class="image image_resized" style="text-align:center;width:75%;"><img src="/uploads/specculative_decoding_flow_5c2970a026.png" alt="specculative-decoding-flow.png" srcset="/uploads/xsmall_specculative_decoding_flow_5c2970a026.png 64w, /uploads/thumbnail_specculative_decoding_flow_5c2970a026.png 161w, /uploads/small_specculative_decoding_flow_5c2970a026.png 500w, /uploads/medium_specculative_decoding_flow_5c2970a026.png 750w, /uploads/large_specculative_decoding_flow_5c2970a026.png 1000w, /uploads/xlarge_specculative_decoding_flow_5c2970a026.png 1920w" sizes="100vw" width="1920"></figure><ol><li>The draft model proposes the next <code>k</code> tokens based on the current context.</li><li>The target model then verifies these tokens in parallel.<ul><li>If it agrees with the draft model’s prediction at position <code>i</code>, the token is accepted.</li><li>If not, the token is rejected, and all later tokens in the draft are discarded.</li></ul></li><li>The target model generates the next token after the last accepted one, and the cycle continues.</li></ol><p>This technique parallelizes the expensive part (i.e. the forward pass) and replaces many slow sequential steps with a single verification, thus reducing ITL. It is helpful to interactive and latency-sensitive applications, such as:</p><ul><li><strong>AI chatbots</strong>: Faster token generation means quicker replies, more natural conversations, and better user experience.</li><li><strong>Code completion tools</strong>: Developers expect instant suggestions. Even a slight delay per token can disrupt flow and reduce usability.</li><li><strong>AI agents and multi-turn reasoning</strong>: Agents often make multiple calls per task. Reducing ITL at each step speeds up decision-making end-to-end.</li><li><strong>Speech-to-text pipelines</strong>: These models decode audio in long sequences. Lower ITL directly improves perceived latency and output speed.</li></ul><h2>Why acceptance rate is important to speculative decoding performance</h2><p>Speculative decoding promises faster LLM inference, but only when it works well. And its effectiveness depends on a critical factor: <strong>how often the target model accepts the draft model’s predictions</strong>.</p><p>This is known as the acceptance rate (α). It is not a fixed number and can vary based on several factors like decoding strategy (e.g., nucleus vs. random sampling) and application domains.</p><ul><li><strong>High α means:</strong><ul><li>More tokens accepted per speculative round</li><li>Fewer target model forward passes</li><li>Lower latency, higher throughput, better GPU utilization</li></ul></li><li><strong>Low α means</strong>:<ul><li>Many draft tokens are rejected</li><li>Time and compute wasted on both generating and verifying unused tokens</li><li>Falling back to sequential decoding more frequently, which defeats the point of speculative acceleration</li></ul></li></ul><h3>Simulate the effect of acceptance rate</h3><p>To understand how acceptance rate impacts performance, we patched <a target="_blank" rel="noopener noreferrer" href="https://github.com/vllm-project/vllm">vLLM</a> to simulate speculative decoding using <strong>theoretical acceptance rates.</strong> We bypassed the draft model and tested different speculative token counts (the number of tokens proposed by the draft model each step).</p><p>Here’s how we set it up:</p><ul><li><strong>Target model</strong>: Llama-3.1-8B-Instruct</li><li><strong>Acceptance rates (α)</strong>: 0.2, 0.4, 0.6, 0.8</li><li><strong>Speculative token counts (γ)</strong>: 3, 5, 7, 9</li><li><strong>Baseline E2E latency (E2EL) without speculative decoding</strong>: 4065.42 ms</li></ul><p>We evaluated the following metrics:</p><ul><li><strong>Output throughput</strong> (Output Tokens Per Second/TPS)</li><li><strong>E2EL</strong> in milliseconds</li><li><strong>Speedup</strong> relative to baseline</li><li><p><strong>Acceptance length (τ)</strong>: The average number of tokens accepted per round of decoding. According to the paper <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/pdf/2211.17192">Fast Inference from Transformers via Speculative Decoding</a>, theoretically, it is calculated with the formula:</p><figure class="image"><img src="/uploads/acceptance_length_formula_spec_decoding_5cbf21f17f.png" alt="acceptance-length-formula-spec-decoding.png" srcset="/uploads/xsmall_acceptance_length_formula_spec_decoding_5cbf21f17f.png 64w, /uploads/thumbnail_acceptance_length_formula_spec_decoding_5cbf21f17f.png 245w, /uploads/small_acceptance_length_formula_spec_decoding_5cbf21f17f.png 500w, /uploads/medium_acceptance_length_formula_spec_decoding_5cbf21f17f.png 750w, /uploads/large_acceptance_length_formula_spec_decoding_5cbf21f17f.png 1000w" sizes="100vw" width="1000"></figure></li></ul><p>Here are the results:</p><figure class="image"><img src="/uploads/simulated_speculative_decoding_d4d5f52a19.png" alt="simulated-speculative-decoding.png" srcset="/uploads/xsmall_simulated_speculative_decoding_d4d5f52a19.png 64w, /uploads/thumbnail_simulated_speculative_decoding_d4d5f52a19.png 157w, /uploads/small_simulated_speculative_decoding_d4d5f52a19.png 500w" sizes="100vw" width="500"></figure><p>Key takeaways:</p><ol><li>The higher the acceptance rate (α), the greater the speedup.</li><li>Speculative token counts (γ) should be increased only when the acceptance length is high. Otherwise, performance may be negatively affected.</li><li>Latency drops and throughput rises almost linearly with α.</li><li>At α ≥ 0.6 and γ ≥ 5, we observed 2–3× speedups over baseline decoding.</li></ol><p>But how do you actually achieve a high acceptance rate in the real world?</p><h2>Why the draft model makes or breaks speedup</h2><p>In the previous section, we showed that speculative decoding can offer up to 3× speedups, but only in theory. Those results were based on simulated acceptance rates, not actual draft model performance.</p><p>To understand how this works in practice, we ran real-world benchmarks using both vLLM and SGLang with and without their respective <strong>EAGLE 3 draft models</strong>.</p><blockquote><p>Note: <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/pdf/2503.01840">EAGLE</a> improves on vanilla speculative decoding. It reuses the top-layer features of the target model (the features before the LM head). It trains the draft model to predict the next feature and then uses the target model’s LM head to obtain the draft token. By leveraging the rich information from the target model, EAGLE achieves significantly better acceleration compared to vanilla speculative sampling.</p></blockquote><p>Here is our test setup:</p><ul><li><strong>Target model</strong>: Llama-3.1-8B-Instruct</li><li><strong>Speculative token count (γ)</strong>: 3</li><li><strong>Datasets</strong>: MT-Bench, HumanEval, CNN/DailyMail</li><li><strong>Baseline</strong>: Regular decoding without EAGLE 3</li><li><strong>Compared against</strong>: EAGLE 3-enabled draft models</li></ul><p>Here are the measured results:</p><figure class="image"><img src="/uploads/speculative_decoding_results_in_practice_7bdf31c614.png" alt="speculative-decoding-results-in-practice.png" srcset="/uploads/xsmall_speculative_decoding_results_in_practice_7bdf31c614.png 64w, /uploads/thumbnail_speculative_decoding_results_in_practice_7bdf31c614.png 166w, /uploads/small_speculative_decoding_results_in_practice_7bdf31c614.png 500w" sizes="100vw" width="500"></figure><p>Despite achieving ~2× speedups, we found that real-world acceptance lengths (τ) were lower than ideal. Based on the test results in the previous section, the acceptance rates (α) were likely in the 0.6–0.8 range, not the near-perfect values used in theory.</p><p>We believe two major factors limit acceptance rates when using out-of-the-box EAGLE 3 draft models:</p><ul><li><strong>Task domain mismatch</strong>: The datasets we used above are generic. If your dataset has specific characteristics, which is often the case in practice, the acceptance rate could be lower and the speedup might be worse.</li><li><strong>Context length mismatch</strong>: EAGLE 3 draft heads were not optimized for long contexts. If your use case involves long prompts, acceptance rates drop due to lack of alignment.</li></ul><p>Based on these findings, our conclusion is:</p><ul><li><strong>To consistently achieve high acceptance rates and up to 3x performance gains, you need to have the right draft model</strong>. For this purpose, you can train a custom draft model on your own data. This ensures it learns the patterns of your specific workload, leading to higher acceptance rates and faster inference.</li><li>However, if your current model already yields a good acceptance rate for your use case, you don’t need extra training. Speculative decoding will still deliver decent speedups.</li></ul><h2>How to train your own draft model</h2><p>To validate our conclusion and test speculative decoding in practice, we trained a custom EAGLE 3 draft model using the official <a target="_blank" rel="noopener noreferrer" href="https://github.com/SafeAILab/EAGLE">EAGLE repository</a>.</p><ul><li><strong>Dataset</strong>: A combined one of UltraChat-200k and ShareGPT</li><li><strong>Hardware</strong>: Training with 8 H200 GPUs</li><li><strong>Result:</strong> Achieved an acceptance rate (α) of 3.02 with 5 speculative tokens (γ = 5)</li></ul><p>We successfully replicated EAGLE 3 and observed the hypothetical speedup in downstream inference benchmarks.</p><p>Here is how we did it.</p><h3>Prepare the dataset</h3><p>The EAGLE repo provides a script to build a training dataset by mixing UltraChat-200k and ShareGPT. Run the <a target="_blank" rel="noopener noreferrer" href="https://github.com/aarnphm/EAGLE/blob/main/eagle/train/eagle3/prepare.py">script</a> to convert both datasets to a shared ShareGPT-style format:</p><pre><code class="language-bash">python eagle/train/eagle3/prepare.py</code></pre><p>Note that:</p><ul><li>The default configuration is used for training on a single GPU. If you have 8 H200 GPUs, set <code>gradient_accumulation_steps = 8</code>. This helps fully utilize available VRAM and accelerates the epoch. If you're using fewer GPUs, adjust this value accordingly.</li><li>For long-context support (e.g. 8k+ tokens), some customization is required in DeepSpeed to ensure proper context window handling.</li><li>Optionally, set up wandb for tracking. If you don’t want to use it, comment out the wandb logic in the training code.</li></ul><h3>Evaluate the trained model</h3><p>Once configured, training can be launched. After 10 epochs, you’ll be ready to validate your draft model.</p><ol><li><p>Copy your trained config into the state directory:</p><pre><code class="language-bash">cp config.json output_dir/state_&lt;epoch&gt;/config.json</code></pre></li><li><p>Run vLLM with EAGLE 3:</p><pre><code class="language-bash">VLLM_USE_V1=1 VLLM_LOGGING_LEVEL=DEBUG \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 8192 \
  -tp 1 \
  --speculative-config '{"method": "eagle3", "model": "path/to/state_dir", "num_speculative_tokens": 5, "draft_tensor_parallel_size": 1}'</code></pre></li><li><p>In a separate terminal, benchmark the setup and here is an example command:</p><pre><code class="language-bash">vllm bench serve \
  --save-result \
  --sharegpt-output-len 1024 \
  --dataset-name sharegpt \
  --endpoint-type openai \
  --ramp-up-strategy linear \
  --ramp-up-start-rps 1 \
  --ramp-up-end-rps 2 \
  --label eagle3-2048-o1024-tp1 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-path ~/path/to/your/datasets \
  --endpoint-type openai-chat \
  --endpoint "/v1/chat/completions"</code></pre></li></ol><h2>Conclusion</h2><p>Speculative decoding is a powerful technique to speed up LLM inference. It’s also a fast-moving research area. Other methods like <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/pdf/2404.16710">LayerSkip</a> and <a target="_blank" rel="noopener noreferrer" href="https://arxiv.org/pdf/2502.09419v1">MTP</a> are being actively explored and may offer further gains in the future.</p><p>Through both simulation and real-world benchmarks, we showed that:</p><ul><li>High acceptance rates are necessary to realizing meaningful gains, often 2–3× in throughput and latency.</li><li>Generic, out-of-the-box draft models fall short when used in domain-specific or long-context workloads.</li><li>Training your own draft model on your own data is helpful to aligning the draft model with your use case and achieving optimal performance.</li></ul><p>If your LLM application is latency-sensitive, don’t just plug in speculative decoding as a drop-in solution. Benchmark it. Tune it. Train your draft model if necessary.</p><p>Still have questions?</p><ul><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/join-slack-from-blog-speculative-decoding">Join our Slack community</a> to learn from others and discuss speculative decoding</li><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/contact-us-from-blog-speculative-decoding">Schedule a call</a> with our inference experts</li><li><a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/">Read our LLM Inference Handbook</a> to learn more inference optimization techniques</li></ul>

Learn what speculative decoding is, how it speeds up LLM inference, why draft model choice matters, and when training your own delivers up to 3× performance gains.

spec-decoding.png

Get 3× Faster LLM Inference with Speculative Decoding Using the Right Draft Model

<p>At Bento, we work closely with enterprises across industries like healthcare, finance, and government. Many of them choose to move LLM workloads into on-prem environments, and for good reasons:</p><ul><li><strong>Data privacy &amp; compliance</strong>. LLMs now power critical applications like AI agents and RAG pipelines. These systems often need direct access to sensitive data, making privacy and compliance non-negotiable. Running inference on-prem keeps both data and models entirely within a controlled environment.</li><li><strong>Consistent and reliable performance</strong>. Dedicated resources ensure low-latency, predictable responses, without relying on external APIs or shared cloud infrastructure. AI teams can fully control, customize, fine-tune, and integrate models with internal systems.</li><li><strong>Cost reduction</strong>. While initial setup requires high investment, on-prem deployments can become far more cost-effective than usage-based cloud APIs in the long run.</li></ul><p>But here’s the catch: setting up an on-prem LLM stack isn’t just about buying GPUs and running vLLM on Kubernetes. The real challenge lies in everything that comes after: scaling workloads, maximizing GPU utilization, optimizing performance, and serving models reliably in production.</p><p>And that’s where most AI teams discover a missing piece in the stack.</p><h2>The modern on-prem stack for LLM deployments</h2><p>To run LLMs on-prem, you need to orchestrate a reliable, scalable, and efficient inference stack, which entails three core layers working together:</p><ol><li><strong>Infrastructure layer</strong>: This is the physical foundation that powers your system, including servers, GPUs, networking, storage, power, and cooling.</li><li><strong>Container orchestration layer</strong>: Platforms like Kubernetes and OpenShift manage containerized workloads, schedule jobs, and balance resource usage across nodes. This layer abstracts the underlying hardware and provides deployment primitives for your system.</li><li><strong>Inference platform layer</strong>: This is the often-overlooked part where things fall short. It:<ul><li>Manages model deployments and iterations</li><li>Optimizes inference performance, with <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/getting-started/choosing-the-right-inference-framework">different runtimes</a> and distributed serving strategies, like <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/prefill-decode-disaggregation">prefill-decode disaggregation</a> and <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/prefix-caching">KV aware routing</a></li><li>Handles autoscaling and optimizes cold starts &amp; resource utilization</li><li>Provides observability for LLM performance metrics and costs</li></ul></li></ol><figure class="image image_resized" style="width:75%;"><img src="/uploads/on_prem_llm_deployment_stack_e54655d0eb.png" alt="on-prem-llm-deployment-stack.png" srcset="/uploads/xsmall_on_prem_llm_deployment_stack_e54655d0eb.png 64w, /uploads/thumbnail_on_prem_llm_deployment_stack_e54655d0eb.png 185w, /uploads/small_on_prem_llm_deployment_stack_e54655d0eb.png 500w, /uploads/medium_on_prem_llm_deployment_stack_e54655d0eb.png 750w, /uploads/large_on_prem_llm_deployment_stack_e54655d0eb.png 1000w" sizes="100vw" width="1000"></figure><p>Most organizations already have layers 1 and 2 in place. Layer 3 matters for long-term scaling but it’s often missing or built ad hoc.</p><h2>Risks of on-prem LLM deployments and how an inference platform can help</h2><p>Moving LLMs on-prem promises better control and cost efficiency, but it also shifts more responsibility to your infrastructure/platform team. Let’s break down the most common pain points and how an inference platform helps solve them.</p><h3>1. Slow time to market</h3><p>LLMs are evolving rapidly. New frontier models and inference optimization techniques are released constantly. As a result, AI teams are under pressure to evaluate and deploy them quickly. But turning experiments into production-ready services often gets bogged down by manual infrastructure work:</p><ul><li>Packaging models and building container images</li><li>Integrating with the right inference backend</li><li>Tuning performance for specific latency/throughput targets</li><li>Setting up autoscaling, observability, and reliability guarantees</li><li>Implementing custom serving logic or <a target="_blank" rel="noopener noreferrer" href="https://docs.bentoml.com/en/latest/get-started/model-composition.html">multi-model chaining</a> for advanced agentic systems</li></ul><p>If a model needs to be optimized for a specific use case, such as distributed inference or benchmarking, <strong>it can add weeks or months to the timeline</strong>.</p><p>AI teams should be able to ship fast, not dealing with infrastructure issues for every model.</p><p>A strong inference platform removes these roadblocks:</p><ul><li>Standardized workflows to deploy and iterate on new models in minutes or hours, not weeks</li><li>SOTA open-source models pre-packaged with different optimization options, deployable with a few clicks</li><li>Built-in support for performance tuning, autoscaling, and observability</li><li>Developer-friendly tools and abstractions for implementing custom logic</li></ul><p>Without this layer, projects get stuck in limbo, which means missed market opportunities. Speed is a competitive edge, and an inference platform helps you move faster.</p><h3>2. Poor cost visibility and GPU utilization</h3><p>Many teams turn to on-prem deployments with the goal of reducing long-term costs. However, those cost savings only materialize if you’re using your compute efficiently.</p><p>On-prem GPUs are a fixed resource. If they’re sitting idle, or worse — running zombie workloads that no one’s monitoring — you’re burning capital without generating value.</p><p>To truly realize the cost benefits of on-prem, you need two things:</p><ul><li><strong>Fast autoscaling to respond to actual traffic patterns</strong>. During spikes, workloads should scale up to handle new requests. When idle, they should scale down so resources can be reused for other tasks to maximize ROI.</li><li><strong>Clear visibility into GPU utilization and costs</strong>. This ensures you know whether your infrastructure is working efficiently and whether you’re getting the savings you expect.</li></ul><p>This is where an inference platform plays a critical role. It provides the autoscaling logic and real-time metrics needed to maximize GPU usage and reduce waste.</p><p>The first two layers weren’t designed to handle GenAI workloads like LLMs. They don’t support fast autoscaling for multi-gigabyte LLM container images, or intelligent scheduling based on KV cache availability (higher cache hit rates means better use of techniques like <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/prefix-caching">prefix caching</a> to reduce compute costs). The inference layer fills that gap.</p><h3>3. Performance bottlenecks</h3><p>As LLM context windows expand and use cases grow more complex, single-node inference optimizations quickly hit a wall. Techniques like continuous batching can help squeeze more out of a single GPU, but they’re no longer enough.</p><p>Modern LLM inference is <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/the-shift-to-distributed-llm-inference">shifting toward distributed architectures</a>:</p><ul><li><strong>Prefill and decode require different resources.</strong> Prefill is compute-intensive, while decode is memory-bandwidth bound and token-by-token. Co-locating both on the same hardware leads to contention and waste. They need separate resources to deliver the optimal performance and scale independently.</li><li><strong>KV caches create memory pressure.</strong> Long-context models or agentic systems rely heavily on techniques like <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/prefix-caching">prefix caching</a> to accelerate inference and reduce costs. However, storing tons of KV caches consumes significant GPU memory.</li><li><strong>Model sizes exceed single-GPU limits.</strong> A 70B parameter model in FP16 needs over 140 GB of memory (not to mention overhead for runtime or KV caches). To serve models at this scale, you need techniques like <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/data-tensor-pipeline-expert-hybrid-parallelism">tensor parallelism and expert parallelism</a> (for MoE models) to split and distribute workloads across multiple GPUs.</li></ul><p>Solving these problems means building a distributed inference stack, including <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/prefill-decode-disaggregation">disaggregated serving</a>, <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/prefix-aware-routing">intelligent routing</a>, and <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/kv-cache-offloading">KV cache offloading</a>. However, most AI teams don’t have the time to solve them from scratch in on-prem environments.</p><p>This is where a modern inference platform becomes essential. Ideally, it should support the latest distributed inference optimizations out of the box and make them accessible to developers, without needing to reinvent complex infrastructure.</p><h3>4. Observability challenges</h3><p>For LLM workloads in production, especially those with strict SLAs and uptime guarantees, you must have deep, inference-specific observability. But traditional monitoring stacks, which are built for web services and microservices, don’t offer the visibility to operate and optimize LLM workloads.</p><p>To ensure reliable inference at scale, you need visibility into LLM-specific metrics and behavior, such as:</p><ul><li><a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/llm-inference-metrics">Time to First Token (TTFT), Time per Output Token (TPOT), and Token Generation Time</a> for understanding latency characteristics across different stages of token generation</li><li>Prefix cache hit rate to measure the effectiveness of KV cache reuse and inform routing decisions</li><li>Cost per request or token to analyze and optimize compute cost per unit of work</li><li>Usage tracking and cost analysis broken down by team or project to ensure accountability and identify optimization opportunities</li><li>Prompt filtering, output anomalies, and model behavior drift to monitor output quality and unexpected behavior</li></ul><p>These metrics don’t live at the infrastructure or orchestration layers; they live at the inference layer. Without them, teams are left guessing where bottlenecks are or how to tune performance.</p><p>A modern inference platform provides LLM-specific observability that helps AI teams optimize performance and maintain reliability over time.</p><h3>5. Operational inefficiency</h3><p>As organizations scale their AI initiatives, <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/what-is-inference-ops">inferenceOps</a> quickly become a major source of complexity.</p><p>It’s no longer just about serving a single LLM. Enterprises today deploy <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/a-guide-to-compound-ai-systems">compound AI systems</a> that combine multiple models: LLMs, embedding models, SLMs, and reasoning models. They power agentic workflows, RAG pipelines, and decision-making tools, which lead to heterogenous AI workflows.</p><p>This also means heterogeneous compute requirements. Some models are GPU-bound, others are CPU- or IO-heavy. Managing these different workloads on a shared cluster requires a carefully orchestrated architecture to avoid bottlenecks and maximize efficiency.</p><p>But here’s the problem: maintaining these systems and compute resources manually is a <strong>massive tax on engineering teams</strong>. It diverts attention from core product development and slows down innovation.</p><p>What’s needed is a standardized layer that abstracts away the complexity. An inference platform fills that gap. It offers:</p><ul><li>Consistent workflows for deploying and updating models</li><li>Tooling for managing multi-model, multi-team workloads</li><li>Built-in scheduling and resource allocation logic to handle heterogeneous pipelines across multiple on-prem clusters</li></ul><p>InferenceOps isn’t just an infrastructure problem; it’s a productivity one. A good inference platform gives teams the tools to ship faster with less friction.</p><h2>Bento On-Prem: The inference platform layer for modern LLM workloads</h2><p>The five risks we outlined above are the biggest blockers for on-prem LLM success. They all stem from one missing piece: the inference platform layer.</p><p>That’s exactly where <strong>Bento On-Prem</strong> comes in.</p><ul><li>Let AI teams move fast with production-ready workflows for deploying, scaling, and managing LLMs.</li><li>Get optimal performance with built-in support for fast autoscaling, distributed serving, and inference-specific observability</li><li>Seamlessly integrate with your existing infrastructure and container orchestration layers</li><li>Keep all data and models fully within your private on-prem environment</li></ul><p>Built for speed, simplicity and control, Bento On-Prem lets AI teams move fast without additional complexity to your stack.</p><p>Running LLMs on-prem? We’d love to help.</p><ul><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/contact-us-from-blog-on-prem-llms">Contact us</a> today to see how Bento On-Prem can help you deploy secure, scalable LLM inference entirely within your control.</li><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/join-slack-from-blog-on-prem-llms">Join our Slack community</a> to connect with other teams and chat about secure, scalable LLM deployments.</li><li>Check out our <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/">LLM Inference Handbook</a> to learn everything about inference.</li></ul>

Learn why an inference platform is the missing layer in on-prem LLM deployments and how it helps optimize inference performance, reduce compute cost, and guarantee reliability.

on-prem-llms.png

Inference Platform: The Missing Layer in On-Prem LLM Deployments

Explore the top open-source embedding models and find answers to some FAQs about them.

embedding-models.png

A Guide to Open-Source Embedding Models

Explore the top open-source VLMs and find answers to some FAQs about them.

vision-language-models.png

Multimodal AI: A Guide to Open-Source Vision Language Models

Explore the top open-source TTS models and find answers to some FAQs about them.

tts-models.png

Exploring the World of Open-Source Text-to-Speech Models

Understand open-source image generation models and find answers to frequently asked questions about them.

image-generation-models.png

A Guide to Open-Source Image Generation Models

<p>You built an LLM-powered application. The demo went great. Leadership is impressed. Applause. Green lights. Your team is ready to ship.</p><p>But fast forward a few weeks:</p><ul><li>Your GPU usage is climbing fast. Costs are spiking. How do you secure enough compute, optimize cost, and ensure reliability?</li><li>You are exploring different distributed inference strategies, but can’t tell which parallelism strategy or inference setup best fits your use case.</li><li>A new frontier open model just dropped. Can you immediately launch a deployment and start experimenting or are you stuck waiting on infrastructure support?</li><li>A model misfires in production. Can you trace, debug, and fix it, without slowing your team down?</li></ul><p>These aren’t edge cases. They’re the new normal when inference scales. The problem isn’t just missing tools. It’s a deeper operational gap.</p><p>That’s where a mindset shift is needed and we call it <strong>InferenceOps</strong>. In this post, we’ll break down what it means, why it’s increasingly important, and how a comprehensive InferenceOps platform helps enterprises build reliable and scalable LLM applications in production.</p><h2>InferenceOps: A mindset shift</h2><p>InferenceOps is a set of best practices and operational principles for running, scaling, and managing inference reliably and efficiently in production.</p><p>It represents a mindset shift: from treating inference as an afterthought to managing it as a critical layer of the modern AI stack. This includes but is not limited to:</p><ul><li>Continuously and rapidly iterating on models and inference code</li><li>Optimizing inference performance and cost across varied workloads</li><li>Ensuring reliability across heterogeneous models and distributed compute environments</li></ul><p>This shift empowers AI teams to move faster and build differentiated AI systems to better serve their customers.</p><figure class="image image_resized" style="width:75%;"><img src="/uploads/what_is_inferenceops_de47b631f8.png" alt="what-is-inferenceops.png" srcset="/uploads/xsmall_what_is_inferenceops_de47b631f8.png 64w, /uploads/thumbnail_what_is_inferenceops_de47b631f8.png 190w, /uploads/small_what_is_inferenceops_de47b631f8.png 500w, /uploads/medium_what_is_inferenceops_de47b631f8.png 750w, /uploads/large_what_is_inferenceops_de47b631f8.png 1000w, /uploads/xlarge_what_is_inferenceops_de47b631f8.png 1920w" sizes="100vw" width="1920"></figure><h2>Why InferenceOps matters now more than ever?</h2><p><strong>AI is no longer optional; it’s eating the world.</strong> Every business that wants to compete today needs AI at its core. While choosing the right model is crucial, it’s your inference layer that actually delivers that model’s value to users. A strategy built solely on third-party LLM APIs can get you started quickly, but it won’t sustain a differentiated, production-grade system.</p><p><strong>Why relying on external LLM APIs falls short:</strong></p><ul><li><strong>Data privacy &amp; compliance:</strong> You can’t send sensitive or regulated data off-premises.</li><li><strong>Noisy neighbor effects:</strong> If another large customer saturates the shared endpoint, your latency and availability suffer.</li><li><strong>No custom or fine-tuned models:</strong> LLM APIs only provide general-purpose models, preventing you from customizing models to your proprietary data and domain.</li><li><strong>Zero control over the model:</strong> You can’t lock in a specific model version, roll back, or meet strict audit requirements.</li><li><strong>Limited performance tuning:</strong> You can’t apply inference optimizations uniquely tailored to your SLAs or workload characteristics, align with your specific latency, throughput, and concurrency requirements.</li><li><strong>Cost inefficiency at scale:</strong> Shared-API pricing and opaque billing make it impossible to optimize per-request costs.</li></ul><p>For a deeper breakdown, see <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/serverless-vs-dedicated-llm-deployments">Serverless vs. Dedicated LLM Deployments: A Cost-Benefit Analysis</a>.</p><p><strong>And that lack of customization hamstrings your ability to build lasting advantage:</strong></p><ol><li><strong>Compound AI systems</strong> are where you win — chaining &amp; composing models into richer workflows. See <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/a-guide-to-compound-ai-systems">A Guide to Compound AI Systems</a> for details.</li><li><strong>Tailored inference stacks</strong> let you architect for precise SLAs and cost targets across different workloads.</li><li><strong>Fine-tuning and custom models</strong> drive domain-specific accuracy and intellectual property capture.</li></ol><p>Ultimately, <strong>inference quality is product quality</strong>. Mission-critical AI systems demand rock-solid performance, predictable costs, and airtight security, which only owning your inference infrastructure can deliver.</p><h2>What are the core pillars of InferenceOps?</h2><p>Like DevOps, InferenceOps isn’t defined by a single tool or role; it’s a set of operational principles. At its core, it shares many familiar foundations with DevOps, including:</p><ul><li>Automation and CI/CD pipelines</li><li>System observability, monitoring and alerting</li><li>Reliability with versioning, rollbacks, and safe deployments</li></ul><p>These common practices are just the baseline. InferenceOps comes with unique requirements, especially for running and scaling LLMs in production.</p><h3>Heterogeneous workflow management</h3><p>In practice, enterprise AI is rarely powered by just one model. Different teams, products, and use cases often require different LLMs, each with distinct configurations and optimizations for optimal performance. One use case may need long-context summarization, and yet another may rely on high-throughput batch processing.</p><p>And it doesn’t stop at LLMs. You’ll often see small language models (SLMs), embedding models, fine-tuned models, and custom models running alongside LLMs in modern AI systems.</p><p>This results in a complex web of inference workflows around LLMs, which can quickly become operational sprawl.</p><p>A unified InferenceOps platform should provide a standardized way to manage diverse inference workflows. That means:</p><ul><li>Coordinating deployments across diverse model types</li><li>Collectively optimizing their compute resources</li><li>Managing different inference setups</li><li>Centralizing visibility into cost, usage, and performance</li></ul><p>As your LLM inference scales, this kind of unification becomes more important. Without it, each new use case adds operational debt. With it, teams can scale responsibly, innovate faster, and maintain control.</p><figure class="image image_resized" style="width:75%;"><img src="/uploads/inferenceops_workflow_9145194132.png" alt="inferenceops-workflow.png" srcset="/uploads/xsmall_inferenceops_workflow_9145194132.png 64w, /uploads/thumbnail_inferenceops_workflow_9145194132.png 245w, /uploads/small_inferenceops_workflow_9145194132.png 500w, /uploads/medium_inferenceops_workflow_9145194132.png 750w, /uploads/large_inferenceops_workflow_9145194132.png 1000w, /uploads/xlarge_inferenceops_workflow_9145194132.png 1920w" sizes="100vw" width="1920"></figure><h3>Unified compute across any infrastructure</h3><p>Different LLMs require different types of compute to deliver optimal performance. Some workloads are compute-bound, others memory-bound. Some benefit from high-throughput batching on A100s, while others demand low-latency responses on edge devices. Many AI teams even combine CPUs, GPUs, and I/O-heavy workloads in the same inference pipeline for complex AI systems.</p><p>To balance availability and cost, as well as to avoid vendor lock-in, AI teams often spread inference workloads across multiple cloud regions/providers and on-prem GPU vendors. This is especially true when dealing with GPU scarcity, region-specific latency requirements, or high GPU pricing to ensure performance and control cost.</p><p>But with flexibility comes complexity. A compute-agnostic InferenceOps platform should enable AI teams to:</p><ul><li>Run, scale, and manage inference seamlessly across hybrid compute environments</li><li>Optimize GPU cost and dynamically match workloads to the most suitable hardware to maximize efficiency</li><li>Avoid vendor lock-in while maintaining velocity and cost efficiency in acquiring and scaling compute</li></ul><h3>Inference workload optimization</h3><p>Running LLM inference isn’t just about launching a model server. It’s an ongoing optimization effort in engineering for better performance and cost-efficiency.</p><p>The first challenge starts at build time. Many models require compilation or runtime optimization before serving, using tools like TensorRT-LLM and TorchScript. Each model architecture comes with its own quirks. Getting this right takes effort and skipping it leaves performance on the table.</p><p>Then there’s scaling complexity. LLM workloads are not like typical web apps. They involve provisioning compute, pulling large container images, and loading model into GPUs, which makes cold starts slow and painful. See <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/scaling-ai-model-deployment">Scaling AI Models Like You Mean It</a> for details.</p><p>As traffic grows and inference workloads get heavier, teams often begin with horizontal scaling, spinning up more GPU instances, but even that is far from enough when you’re serving large models across a multi-GPU cluster. That’s why leading AI teams are moving towards <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/blog/the-shift-to-distributed-llm-inference">distributed LLM inference strategies</a>, such as:</p><ul><li>Prefill-decode disaggregation</li><li>Prefix and KV cache aware routing</li><li>KV cache management</li><li>Tensor, pipeline, and expert parallelism techniques</li></ul><p>They bring better resource allocation, smarter GPU usage, lower token latency, and reduced cost per token.</p><p>However, distributed LLM inference is complex by nature. Simply spinning up more Pods on Kubernetes won’t cut it. You need purpose-built infrastructure for distributed LLM workloads. Done poorly, it leads to an over-provisioned, under-optimized system, wasting expensive GPUs while delivering sub-par performance.</p><p>A developer-friendly InferenceOps platform should abstract away the complexity and provide out-of-the-box support for these advanced optimizations.</p><h3>Fast iteration without compromising reliability</h3><p>From real-time chatbots to coding assistants, LLM inference now sits on the hot path of user-facing systems. That means production-grade reliability isn’t optional. You need:</p><ul><li>Strict SLAs and uptime guarantees, especially for latency-sensitive workloads</li><li>LLM-specific observability, going beyond generic system metrics to include:<ul><li>Time to First Token (TTFT) and Time per Output Token (TPOT)</li><li>Tokens per Second (TPS) and Requests per Second (RPS)</li><li>Cost per request/token</li><li>Prompt filtering, output anomalies, model behavior drift</li></ul></li><li>Robust and automated pipelines, so your team can ship new models and code updates with zero downtime</li></ul><p>But reliability often slows down velocity. Building this kind of infrastructure from scratch is slow, expensive, and error-prone. It requires specialized knowledge in GPU orchestration, observability stacks, and distributed inference systems. This means engineers have to spend more time fighting infrastructure than experimenting or shipping new ideas.</p><p>A comprehensive InferenceOps platform resolves this tension by providing:</p><ul><li>Native support for LLM-specific observability and monitoring</li><li>Easy way to launch benchmark load testing to identify performance bottlenecks</li><li>Optimized CI/CD for updates to models, adapters, and inference logic</li><li>Built-in safeguards for deployments, rollbacks, and canary releases</li></ul><p>With operational burdens offloaded, AI teams can move faster without compromising efficiency, reliability, and security.</p><h2>Conclusion</h2><p>InferenceOps is emerging as a critical foundation for enterprise LLM strategy. With the right practices and platform in place, you can:</p><ul><li>Unify management of diverse models and inference workflows</li><li>Run and scale inference reliably across heterogeneous compute environments</li><li>Continuously optimize performance and cost with workload-specific techniques</li><li>Accelerate iteration from development to production without sacrificing reliability</li></ul><p>At <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/">Bento</a>, we help enterprises run LLMs on their terms. With a distributed architecture and tailored inference optimization, our Bento Inference Platform enables you to deploy, scale, and manage any LLM from a single, unified interface.</p><ul><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/contact-us-from-blog-what-is-inference-ops">Talk to us</a> about bringing InferenceOps into your LLM strategy</li><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/join-slack-from-blog-what-is-inference-ops">Join our Slack community</a> to connect with others building custom LLM applications</li><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/signup-from-blog-what-is-inference-ops">Sign up</a> to try the Bento Inference Platform and start running production-grade inference today</li><li>Learn more about inference in our <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/">LLM Inference Handbook</a></li></ul>

Learn what InferenceOps is, why it matters, and how leading AI teams scale, optimize, and manage LLM inference for production-grade performance and reliability.

inferenceops-blog-cover-image.jpg

What is InferenceOps?

Fog Dong

<p>At Bento, we work to help AI teams deploy and scale models with tailored optimization. That means giving them the ability to tune performance for their specific use cases with ease and speed, so they can achieve better price-performance ratios.</p><p>As part of that effort, we’re excited to introduce <a target="_blank" rel="noopener noreferrer" href="https://github.com/bentoml/llm-optimizer">llm-optimizer</a>, an open-source tool for benchmarking and optimizing LLM inference. It works across multiple inference frameworks and supports any open-source LLM.</p><p>Unlike traditional benchmarking tools that only generate raw numbers, llm-optimizer allows you to define constraints, such as “TTFT under 200ms” or “P99 ITL below 10ms.” This makes it easy to quickly identify the configurations that meet your specific requirements without endless trial and error.</p><p>At the same time, we’re launching the <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/llm-perf/">LLM Performance Explorer</a>, a companion website powered by llm-optimizer. It lets you browse LLM benchmark results directly without running experiments yourself. You can compare different configurations, apply constraints, and quickly see which setup works best for your specific use case.</p><figure class="image"><img></figure><figure class="image"><img src="/uploads/llm_perf_explorer_7131fbb216.png" alt="llm-perf-explorer.png" srcset="/uploads/xsmall_llm_perf_explorer_7131fbb216.png 64w, /uploads/thumbnail_llm_perf_explorer_7131fbb216.png 245w, /uploads/small_llm_perf_explorer_7131fbb216.png 500w, /uploads/medium_llm_perf_explorer_7131fbb216.png 750w, /uploads/large_llm_perf_explorer_7131fbb216.png 1000w, /uploads/xlarge_llm_perf_explorer_7131fbb216.png 1920w" sizes="100vw" width="1920"></figure><h2>Motivation</h2><p>When it comes to self-hosting LLMs, performance tuning can be very tricky. There are so many factors to balance:</p><ul><li><strong>Server&nbsp;parameters</strong>: <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/data-tensor-pipeline-expert-hybrid-parallelism">Parallelism strategies (e.g., data, tensor, pipeline)</a>, batch sizes, <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/kv-cache-offloading">caching</a> strategies, etc.</li><li><strong>Client parameters</strong>: Request rates, concurrency&nbsp;levels,&nbsp;number of prompts, etc.</li><li><strong>Framework&nbsp;differences</strong>: Each&nbsp;<a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/getting-started/choosing-the-right-inference-framework">inference&nbsp;framework</a>&nbsp;(vLLM,&nbsp;SGLang, TensorRT-LLM, etc.) has&nbsp;its own optimization&nbsp;strategies and tuning knobs.</li><li><strong>Workload variations</strong>: Different models, input&nbsp;lengths, and request patterns require different optimizations.</li></ul><p>For most AI teams, the only way to figure it out has been trial and error: endless tests, parameter tweaks, and results that are hard to compare. It’s tedious, time-consuming, and often error-prone.</p><p>This led us to build llm-optimizer. We want to provide engineers a systematic way to test configurations, and give them a clear view of what really works with just a few commands. More importantly, it must support constraint filtering, so that engineers don’t need to sift through mountains of raw result data. This means they can focus on the right parameters for their specific optimization goals. Ultimately, engineers spend less time tuning and more time building.</p><h2>Key features</h2><p>llm-optimizer was built to take the tedious manual work out of LLM performance optimization. You can run structured experiments, apply constraints, and visualize results in one place - all with a few commands.</p><h3>Systematic parameter testing</h3><p>llm-optimizer supports systematic tests of different combinations of server and client parameters across multiple inference frameworks. It works with any open-source LLM. Simply provide the Hugging Face model tag.</p><p>Here is an example with vLLM:</p><pre><code class="language-bash">llm-optimizer \
  --framework vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --server-args "tensor_parallel_size=[1,2];max_num_batched_tokens=[4096,8192]" \
  --client-args "max_concurrency=[32,64];num_prompts=1000;dataset_name=sharegpt" \
  --output-json vllm_results.json</code></pre><p>Parameters like <code>tensor_parallel_size</code> and <code>max_num_batched_tokens</code> come directly from the respective frameworks (currently support vLLM and SGLang). If you’re familiar with other native parameters, you can add them as well.</p><p>Here is another example with SGLang:</p><pre><code class="language-bash">llm-optimizer \
  --framework sglang \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --server-args "tp_size*dp_size=[(1,4),(2,2),(4,1)];chunked_prefill_size=[2048,4096]" \
  --client-args "max_concurrency=[32,64,128];num_prompts=[500,1000]" \
  --output-dir optimization_results \
  --output-json results.json \
  --continue \
  --rest 10 \
  --warmup-requests 3</code></pre><p>This command generates <strong>36 unique test runs</strong>:</p><ul><li>3 tensor/data parallelism combinations × 2 prefill sizes = 6 server configs</li><li>3 concurrency values × 2 prompt counts = 6 client configs</li><li>6 × 6 = 36 total configurations, each with 3 warmup requests and a 10-second rest between runs</li></ul><p>This means 36 results you can analyze or filter further using constraints.</p><h3>Constraint settings</h3><p>Not every benchmark result is useful. What really matters is finding the configuration that meets your performance goals, especially when you have specific SLOs for inference. Without filtering, large test runs can generate too much noise and make it harder to analyze results.</p><p>With llm-optimizer, you can:</p><ul><li><strong>Filter results</strong> to only include configurations that meet your requirements</li><li><strong>Identify optimal setups</strong> for latency or throughput targets</li></ul><p>The following is a typical use case where you can optimize for chatbots or other low-latency workloads.</p><pre><code class="language-bash">llm-optimizer \
  --framework vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --server-args "tensor_parallel_size*data_parallel_size=[(1,2),(2,1)];max_num_seqs=[16,32,64]" \
  --client-args "max_concurrency=[8,16,32];num_prompts=500" \
  --constraints "ttft&lt;200ms;itl:p99&lt;10ms" \
  --output-json chat_optimized.json</code></pre><p>This command will only return results where:</p><ul><li>Time to First Token (TTFT) is below&nbsp;200ms</li><li>99% of Inter-Token Latency (ITL)&nbsp;are below 10ms</li></ul><h3>Comprehensive benchmarking results</h3><p>Benchmarks are only useful if the results tell you something actionable. llm-optimizer provides detailed inference-specific metrics, giving you a clear view of how your model behaves under different conditions.</p><p>For LLMs, performance is best measured at the token level. <a target="_blank" rel="noopener noreferrer" href="https://bentoml.com/llm/inference-optimization/llm-inference-metrics">Key metrics</a> include:</p><ul><li><strong>Time to First Token (TTFT):</strong> How long it takes for the first response token to appear after a request is sent</li><li><strong>Inter-Token Latency (ITL):</strong> The average time between tokens in a response.</li><li><strong>Input/Output Token Throughput:</strong> How many tokens are processed/generated per second during the benchmark.</li></ul><p>In addition, you also get traditional request-level metrics such as end-to-end latency, request throughput, and optimal concurrency. These measurements combined help you make trade-offs for your performance goals.</p><p>Results are output in JSON, easy to filter and compare. Here’s a simplified example:</p><pre><code class="language-yaml language-json">[
  {
    "config": {
      "client": {
        "max_concurrency": 8,
        "num_prompts": 1000,
        "dataset_name": "sharegpt",
        "sharegpt_output_len": 256
      },
      "server": {
        "tensor_parallel_size": 1,
        "max_num_batched_tokens": 4096,
        "max_num_seqs": 16
      },
      "server_args": [
        "--tensor-parallel-size=1",
        "--max-num-batched-tokens=4096",
        "--max-num-seqs=16"
      ]
    },
    "results": {
      "backend": "vllm",
      "dataset_name": "sharegpt",
      "max_concurrency": 8,
      "total_input_tokens": 296523,
      "total_output_tokens": 256000,
      "total_output_tokens_retokenized": 255910,
      "request_throughput": 2.323720163971585,
      "input_throughput": 689.0364741813463,
      "output_throughput": 594.8723619767258,
      "mean_e2e_latency_ms": 3441.463320413,
      "median_e2e_latency_ms": 3413.3226294999304,
      "std_e2e_latency_ms": 130.47813647173385,
      "p99_e2e_latency_ms": 4003.9951376500244,
			...
    },
    "cmd": "vllm serve meta-llama/Llama-3.1-8B-Instruct --host 127.0.0.1 --port 8000 --tensor-parallel-size=1 --max-num-batched-tokens=4096 --max-num-seqs=16"
  },
  
// More results below</code></pre><h3>Performance estimation</h3><p>Sometimes you want to understand how a model will perform before running a full benchmark. Maybe you don’t have the required hardware, your time is limited, or you just want a quick estimate of what to expect. llm-optimizer supports performance estimation that analyzes a model and generates theoretical results without running live tests.</p><p>Here is an example:</p><pre><code class="language-bash">llm-optimizer estimate \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --input-len 1024 \
  --output-len 512 \
  --gpu A100 \
  --num-gpus 2</code></pre><p>This command:</p><ul><li>Analyzes the 8B Llama model on 2x A100 GPUs</li><li>Estimates performance for 1024 input + 512 output tokens</li><li>Shows best latency and throughput configurations</li><li>Provides roofline analysis and identifies bottlenecks</li><li>Generates the commands to run actual benchmarks with SGLang and vLLM</li></ul><p>Expected output:</p><pre><code class="language-bash">=== Configuration ===
Model: meta-llama/Llama-3.1-8B-Instruct
GPU: 2x A100
Precision: fp16
Input/Output: 1024/512 tokens
Target: throughput

Fetching model configuration...
Model: 8029995008.0B parameters, 32 layers

=== Performance Analysis ===
Best Latency (concurrency=1):
  TTFT: 43.1 ms
  ITL: 2.6 ms
  E2E: 1.39 s

Best Throughput (concurrency=512):
  Output: 18873.3 tokens/s
  Input: 23767.8 tokens/s
  Requests: 14.24 req/s
  Bottleneck: Memory

=== Roofline Analysis ===
Hardware Ops/Byte Ratio: 142.5 ops/byte
Prefill Arithmetic Intensity: 52205.5 ops/byte
Decode Arithmetic Intensity: 50.9 ops/byte
Prefill Phase: Compute Bound
Decode Phase: Memory Bound

=== Concurrency Analysis ===
KV Cache Memory Limit: 688 concurrent requests
Prefill Compute Limit: 8 concurrent requests
Decode Capacity Limit: 13 concurrent requests
Theoretical Overall Limit: 8 concurrent requests
Empirical Optimal Concurrency: 16 concurrent requests

=== Tuning Commands ===

--- SGLANG ---
Simple (concurrency + TP/DP):
  llm-optimizer --framework sglang --model meta-llama/Llama-3.1-8B-Instruct --gpus 2 --host 127.0.0.1 --server-args "tp_size*dp_size=[(1, 2), (2, 1)]" --client-args "num_prompts=1000;dataset_name=sharegpt;random_input=1024;random_output=512;num_prompts=1000;max_concurrency=[256, 512, 768]" --output-dir tuning_results --output-json tuning_results/config_1_sglang.json
Advanced (additional parameters):
  llm-optimizer --framework sglang --model meta-llama/Llama-3.1-8B-Instruct --gpus 2 --host 127.0.0.1 --server-args "tp_size*dp_size=[(1, 2), (2, 1)];chunked_prefill_size=[1434, 2048, 2662];schedule_conservativeness=[0.3, 0.6, 1.0];schedule_policy=fcfs" --client-args "num_prompts=1000;dataset_name=sharegpt;random_input=1024;random_output=512;num_prompts=1000;max_concurrency=[256, 512, 768]" --output-dir tuning_results --output-json tuning_results/config_1_sglang.json

--- VLLM ---
Simple (concurrency + TP/DP):
  llm-optimizer --framework vllm --model meta-llama/Llama-3.1-8B-Instruct --gpus 2 --host 127.0.0.1 --server-args "tensor_parallel_size*data_parallel_size=[(1, 2), (2, 1)]" --client-args "num_prompts=1000;dataset_name=sharegpt;random_input=1024;random_output=512;num_prompts=1000;max_concurrency=[256, 512, 768]" --output-dir tuning_results --output-json tuning_results/config_1_vllm.json
Advanced (additional parameters):
  llm-optimizer --framework vllm --model meta-llama/Llama-3.1-8B-Instruct --gpus 2 --host 127.0.0.1 --server-args "tensor_parallel_size*data_parallel_size=[(1, 2), (2, 1)];max_num_batched_tokens=[1024, 1177, 1331]" --client-args "num_prompts=1000;dataset_name=sharegpt;random_input=1024;random_output=512;num_prompts=1000;max_concurrency=[256, 512, 768]" --output-dir tuning_results --output-json tuning_results/config_1_vllm.json</code></pre><p>If you’re not sure what parameters to provide, you can use the interactive mode. It walks you through the setup step by step and generates the same results as if you had provided them directly:</p><pre><code class="language-bash">$ llm-optimizer estimate --interactive

=== LLM Performance Estimation (Interactive Mode) ===

🤖 Model Selection
Popular options: meta-llama/Llama-3.2-1B, meta-llama/Meta-Llama-3-8B, meta-llama/Meta-Llama-3-70B
HuggingFace model ID: meta-llama/Llama-3.2-1B

📏 Sequence Length Configuration
Typical values: 512 (short), 1024 (medium), 2048 (long), 4096 (very long)
Input sequence length (tokens) [1024]: 512
Output sequence length (tokens) [512]: 512

## More options...</code></pre><figure class="image"><img src="/uploads/interactive_demo_14433d2ba1.gif" alt="interactive-demo.gif" srcset="/uploads/xsmall_interactive_demo_14433d2ba1.gif 64w, /uploads/thumbnail_interactive_demo_14433d2ba1.gif 245w, /uploads/small_interactive_demo_14433d2ba1.gif 500w, /uploads/medium_interactive_demo_14433d2ba1.gif 750w, /uploads/large_interactive_demo_14433d2ba1.gif 1000w" sizes="100vw" width="1000"></figure><h3>Interactive user interface</h3><p>Raw numbers are useful, but they’re much easier to interpret when you can see patterns and trade-offs. llm-optimizer offers a visualization tool that turns benchmark results into interactive dashboards.</p><p>After running benchmarks, you don’t have to parse JSON by hand. Simply launch a local dashboard and explore your results visually:</p><pre><code class="language-bash"># Visualize results with Pareto frontier analysis
llm-optimizer visualize --data-file results.json --port 8080

# Combine multiple result files
llm-optimizer visualize --data-file "sglang_results.json,vllm_results.json" --port 8080</code></pre><p>The dashboard will be served at <code>http://localhost:8080/pareto_llm_dashboard.html</code>.</p><blockquote><p>Note: <span style="background-color:rgb(255,255,255);color:rgb(31,35,40);font-family:-apple-system, &quot;system-ui&quot;, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;;font-size:16px;"><span style="-webkit-text-stroke-width:0px;display:inline !important;float:none;font-style:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-weight:400;letter-spacing:normal;orphans:2;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-decoration-thickness:initial;text-indent:0px;text-transform:none;white-space:normal;widows:2;word-spacing:0px;">This feature is still experimental, and we’ll continue improving it in the coming days. For visualized results, check out the </span></span><a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/llm-perf/"><a style="-webkit-text-stroke-width:0px;background-color:rgb(255, 255, 255);box-sizing:border-box;color:rgb(9, 105, 218);font-family:-apple-system, &quot;system-ui&quot;, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;;font-size:16px;font-style:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-weight:400;letter-spacing:normal;orphans:2;text-align:start;text-indent:0px;text-transform:none;text-underline-offset:0.2rem;white-space:normal;widows:2;word-spacing:0px;" rel="nofollow"><u>LLM Performance Explorer</u></a></a><span style="background-color:rgb(255,255,255);color:rgb(31,35,40);font-family:-apple-system, &quot;system-ui&quot;, &quot;Segoe UI&quot;, &quot;Noto Sans&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;;font-size:16px;"><span style="-webkit-text-stroke-width:0px;display:inline !important;float:none;font-style:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-weight:400;letter-spacing:normal;orphans:2;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-decoration-thickness:initial;text-indent:0px;text-transform:none;white-space:normal;widows:2;word-spacing:0px;">.</span></span></p></blockquote><h2>Getting started</h2><p>Install llm-optimizer:</p><pre><code class="language-bash">git clone https://github.com/bentoml/llm-optimizer.git
pip install -e .</code></pre><p>Run a quick estimation:</p><pre><code class="language-bash">llm-optimizer estimate \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --gpu A100 \
  --num-gpus 4 \
  --input-len 1024 \
  --output-len 512</code></pre><p>Run your first benchmark (make sure you have enough GPU resources):</p><pre><code class="language-bash">llm-optimizer \
  --framework vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --server-args "tensor_parallel_size*data_parallel_size=[(1,4),(2,2),(4,1)]";max_num_seqs=[16,32,64]" \
  --client-args "max_concurrency=[8,16,32];num_prompts=500" \
  --constraints "ttft&lt;200ms;itl:p99&lt;10ms" \
  --output-json latency_optimized.json</code></pre><p>LLM inference optimization doesn’t have to mean endless trial and error. llm-optimizer makes it easy to benchmark different configurations, apply constraints, estimate performance, and visualize results — all with a few commands.</p><p>More resources:</p><ul><li>View the benchmark results of top open-source LLMs on the <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/llm-perf/">LLM Performance Explorer</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/join-slack-from-blog-announcing-llm-optimizer">Join our Slack community</a> to connect with other AI engineers and share feedback</li><li>Read our <a target="_blank" rel="noopener noreferrer" href="https://www.bentoml.com/llm/">LLM Inference Handbook</a> for deeper dives into performance optimization</li><li><a target="_blank" rel="noopener noreferrer" href="https://l.bentoml.com/contact-us-from-blog-announcing-llm-optimizer">Schedule a call</a> with our experts to discuss your use case and performance goals</li></ul>

Benchmark and optimize LLM inference performance with SLO constraints across frameworks like vLLM and SGLang.

large_llm-optimizer-cover-image.png

medium_llm-optimizer-cover-image.png

small_llm-optimizer-cover-image.png

thumbnail_llm-optimizer-cover-image.png

xsmall_llm-optimizer-cover-image.png

llm-optimizer-cover-image.png

llm-optimizer: An Open-Source Tool for LLM Inference Benchmarking and Performance Optimization

📢 Introducing [llm-optimizer](/blog/announcing-llm-optimizer) & [LLM Performance Explorer](/llm-perf/) — benchmark and optimize LLM inference with ease.

Bento Blog

llm-optimizer: An Open-Source Tool for LLM Inference Benchmarking and Performance Optimization