LLM inference basics

LLM inference is where models meet the real world. It powers everything from instant chat replies to code generation, and directly impacts latency, cost, and user experience. Understanding how inference works is the first step toward building smarter, faster, and more reliable AI applications.

📄️ What is LLM inference?

LLM inference is the process of using a trained language model to generate responses or predictions based on prompts.

📄️ What is the difference between LLM training and inference?

LLM training builds the model while LLM inference applies it to generate real-time outputs from new inputs.

📄️ How does LLM inference work?

Learn how LLM inference works, from tokenization to prefill and decode stages, with tips on performance, KV caching, and optimization strategies.

📄️ Where is LLM inference run?

Learn the differences between CPUs, GPUs, and TPUs and where you can deploy them.

📄️ Serverless vs. Self-hosted LLM inference

Understand the differences between serverless LLM APIs and self-hosted LLM deployments.

📄️ OpenAI-compatible API

An OpenAI-compatible API implements the same request and response formats as OpenAI's official API, allowing developers to switch between different models without changing existing code.