Understanding LLM Costs

Tokenization

Why your LLM sees 'ChatGPT' as 3 tokens, not 1 word

Your LLM doesn't read words. It reads tokens — subword fragments that determine what your model can understand, how much it costs per query, and why it sometimes misspells simple words.

Context Windows

The budget your model runs on

The context window is the total number of tokens your model can process in one call — input + output combined. It's not just a limit, it's a budget: every token you spend on instructions is a token you can't spend on reasoning.

Attention

Why your inference costs scale quadratically

Attention is the mechanism that lets each token look at every other token to understand context. It's why 'bank' means different things in 'river bank' vs 'bank account' — and why doubling your context length quadruples your compute cost.

Mixture of Experts (MoE)

Why Mixtral is cheaper than GPT-4

MoE models have many specialized 'expert' sub-networks but only activate a few per token. A 47B parameter MoE model might only use 13B parameters per forward pass — giving you large-model quality at small-model cost.

Inference Optimization

Making models fast and cheap

Inference optimization covers quantization (reducing precision), batching (processing multiple requests), KV caching (avoiding recomputation), and speculative decoding (using a small model to draft, a large model to verify). The goal: same quality, less compute.