AI Engineering

Context Windows and Token Budgets for LLMs

What a token is, why the context window is the only memory an LLM has, and how to budget, trim and cache prompts so long conversations stay fast and cheap.

Intermediate·13 min read·Updated Oct 7, 2026

An LLM (large language model) reads and writes tokens, subword pieces produced by a tokenizer, and every call has a context window: a hard limit on input plus output tokens. The model keeps nothing between calls, so the system prompt, tool definitions, retrieved documents, the whole chat history and the space for the answer must all fit in that one window, every time. Cost, latency and answer quality all grow or degrade with what you put in it, which makes the prompt a budget you manage, not a box you fill.

Context

Language models never see characters or words. Before training, a tokenizer is built from a large text corpus, usually with BPE (byte-pair encoding, adapted for neural translation by Sennrich and colleagues in 2016) or a close relative: it starts from bytes and repeatedly merges the most frequent pairs into new vocabulary entries, so common words become one token and rare words, names and code break into several. The model is trained on sequences of these token IDs, and the length of sequence it can attend to at once is its context window. That number has grown fast: models around 2020 accepted roughly 2,000 tokens, windows of tens of thousands appeared in 2023, and as of 2025 windows of several hundred thousand to around a million tokens are common.

You have met tokens on every LLM bill and rate limit (priced and throttled per token, not per request), in the max_tokens parameter, in a "context length exceeded" error after a long chat, and whenever a chatbot "forgot" something said an hour earlier. The simplest way to see the mechanism is a chat client: it has no memory on the server, it just resends the conversation.

chat-turn.ts

// Turn 3 of a chat. The model sees only this array, nothing else.
const messages = [
  {role: 'system', content: 'You are a support assistant for Acme.'},
  {role: 'user', content: 'My order #1042 has not arrived.'},
  {role: 'assistant', content: 'Sorry about that. Which address?'},
  {role: 'user', content: '12 Baker Street. Can you resend it?'},
]
// every token above is billed again on this call, and the answer
// must fit in whatever is left of the window

Token: The unit a model reads and writes: a whole word, part of a word, a punctuation mark or a byte sequence, mapped to an integer ID.
Tokenizer: The fixed algorithm and vocabulary that turns text into token IDs and back. Each model family has its own, so counts differ between providers.
Context window: The maximum number of tokens one call can hold, input and output together.
Max output tokens: The cap you set on the answer (max_tokens or similar). It is reserved out of the window and generation stops when it is reached.
KV cache (key-value cache): Per-token intermediate results the model keeps while generating so it does not recompute the whole prompt for each new token. Its memory grows with prompt length.
TTFT (time to first token): Latency until the first output token appears, dominated by processing the prompt.

Why it matters

Almost every production problem with LLM features has a token cause. Bills that grow faster than usage come from resending ever-longer histories. Slow first responses come from huge prompts. A JSON answer that fails to parse was cut off at the output limit. An assistant that ignores its instructions after a long session had them trimmed away, or buried in the middle of a giant prompt where models pay least attention. None of these show up as an exception unless you check for them, so the budget has to be designed, measured and enforced in code.

What a token is, and why counts surprise you

Because the vocabulary is learned from frequency, token counts depend on how common the text looked in the tokenizer's training data. Ordinary English prose is cheap; code, JSON, numbers, random IDs and many non-English languages split into more, smaller pieces. The rule of thumb of about four characters per English token is fine for a rough estimate and wrong for everything else, so count with the model's own tokenizer or the provider's token-counting endpoint before you rely on a number.

English text

Tokenizers split words into pieces6 tokens for 33 characters

Code

const id = '7f3a9c';punctuation and IDs cost extra

An illustrative split. Common words are one token, a less common word breaks into pieces, and in code most symbols are tokens of their own. Real splits depend on the tokenizer.

Text	Rough token cost	Why
English prose	About 1 token per 4 characters, or 0.75 words	Common words are single vocabulary entries
Code and JSON	Noticeably more per character than prose	Braces, quotes, indentation and identifiers split
Non-English	Often 1.5-3× English for the same meaning	Vocabularies are trained mostly on English; varies a lot by tokenizer
Numbers, IDs	Many small tokens	Digit runs, hashes and base64 have no frequent merges

The window is everything the model sees

A request is a budget split between fixed parts (system prompt, tool definitions), variable parts (retrieved documents, conversation history, the new message) and a reservation for the answer. Most APIs reject a request whose input plus maximum output exceeds the window, or silently shorten the answer, so the reservation is not optional. And the budget you choose is usually smaller than the model's maximum, because every extra token costs money and latency and can dilute the parts that matter.

One request against a 28,000-token budget chosen by the application (illustrative numbers). The answer's space is reserved up front; history and retrieved documents are the parts you trim when the total does not fit.

Why every turn costs more than the last

1
Turn 1 sends the system prompt (1,500 tokens) and one message: about 2,000 input tokens.
2
Each turn adds roughly 500 tokens (question plus answer) to the history, and the whole history is sent again with the next question.
3
Turn 20 therefore sends about 1,500 + 20 × 500 = 11,500 input tokens for a single short question.
4
Summed over the chat, input tokens grow with the square of the number of turns: about 135,000 input tokens for 20 turns, while the user typed perhaps 5,000.

cost.ts

// Illustrative rates only. Use your provider's current pricing.
const RATE_IN = 2 / 1_000_000    // $ per input token (made up)
const RATE_OUT = 8 / 1_000_000   // $ per output token (made up)

const turnCost = (inputTokens: number, outputTokens: number) =>
  inputTokens * RATE_IN + outputTokens * RATE_OUT

// 20-turn chat: system + growing history in, ~300 tokens out
let total = 0
for (let turn = 1; turn <= 20; turn++) {
  total += turnCost(1_500 + turn * 500, 300)
}
console.log(total.toFixed(3))   // 0.318 with these made-up rates

Longer is slower, pricier, and not always better

Before the first output token appears, the model processes the whole prompt (the prefill). In the original Transformer design every token attends to every other, so that work grows with the square of the prompt length (Vaswani et al., 2017). Optimised attention kernels make it much cheaper in practice, but TTFT still rises with prompt size, and the KV cache that generation needs grows with every token in the window, which is why providers price, rate-limit and cap long prompts.

Quality does not scale with length either. Liu et al. (2023) found that models answered best when the relevant passage sat at the beginning or end of a long context and noticeably worse when it was in the middle. Newer models do better on such tests, but an advertised window is a capacity, not a promise that every token in it is used well. Simple "needle in a haystack" tests measure whether a fact can be found, not whether the model reasons across scattered pieces, so test long-context behaviour on your own tasks.

Put it	What	Why
First	System instructions, tool definitions, static reference material	Stable prefix: reusable by prompt caching, and the start of the context gets strong attention
Middle	Older history, supporting documents	The part most likely to be under-used; keep it short and relevant
Last	The current question, the most relevant retrieved passages, a reminder of the output format	The end of the context is read most reliably and is what the answer continues from

Fitting a conversation into the budget

When the parts do not fit, something has to go, and what you drop decides what the model forgets. Drop whole messages, never half of one, and never separate a tool call from its result: many APIs reject a tool result whose call is missing. The system prompt and the new message are fixed; history and retrieved context are the flexible parts.

Strategy	How	Costs you
Drop oldest turns	Keep the newest messages that fit the budget	Forgets early facts and decisions abruptly
Summarise	Fold dropped turns into a running summary message	An extra model call; details get lost in compression
Retrieve	Store history and documents outside; fetch only relevant parts per turn	A retrieval pipeline, and misses when search misses
Trim tool output	Return only the fields the model needs, cap long results	Must design tools for brevity
External state	Keep structured facts (cart, user profile, task list) in your app and inject a compact view	More application code; the most reliable memory

budget.ts

export type Msg = {role: 'system' | 'user' | 'assistant'
  content: string}

// Use the provider's tokenizer or token-count endpoint here.
declare function countTokens(msg: Msg): number

const BUDGET = 28_000          // what you choose, at most the window
const RESERVED_OUTPUT = 4_000  // must match max_tokens on the request

export function fitHistory(system: Msg, history: Msg[], next: Msg) {
  let left = BUDGET - RESERVED_OUTPUT
    - countTokens(system) - countTokens(next)
  const kept: Msg[] = []
  // newest to oldest, whole messages only
  for (let i = history.length - 1; i >= 0; i--) {
    const cost = countTokens(history[i])
    if (cost > left) break
    left -= cost
    kept.unshift(history[i])
  }
  const dropped = history.slice(0, history.length - kept.length)
  return {messages: [system, ...kept, next], dropped}
}

summarize.ts

import type {Msg} from './budget'

declare function complete(prompt: string, maxTokens: number)
  : Promise<string>

// Fold the turns fitHistory dropped into a running summary that is
// sent as one short message near the top of the next request.
export async function compact(summary: string, dropped: Msg[]) {
  if (dropped.length === 0) return summary
  const turns = dropped.map(m => `${m.role}: ${m.content}`).join('\n')
  return complete(
    'Update this conversation summary. Keep facts, decisions, open ' +
      'questions and user preferences. At most 250 words.\n\n' +
      `Summary so far:\n${summary}\n\nNew turns:\n${turns}`,
    500,
  )
}

Pitfalls

Budgeting in characters or words
The four-characters-per-token rule holds only for English prose. Code, JSON, IDs and other languages can cost several times more, so a prompt that "fits" by character count overflows in production. Count tokens with the model's tokenizer, including tool definitions and per-message overhead.
Not reserving space for the answer
Filling the window with input leaves the model too little room to answer, so output stops at the limit: truncated JSON, half a list, a missing closing step. Reserve the output budget first and check the stop reason on every response.
Stuffing the window because it is large
Pasting whole documents into a huge window costs more per call, raises time to first token and buries the relevant part in the middle where models attend least. Retrieve the relevant passages instead and put them near the question.
Changing content at the start of the prompt
A timestamp, request ID or user-specific line at the top of the system prompt makes every prefix unique, so prompt caching never hits and you pay full price for the same instructions on every call. Keep the stable parts first and the per-request parts last.
Trimming the wrong things
Naive truncation can drop the system prompt, cut a message in half, or keep a tool result while dropping the call that produced it, which many APIs reject and which confuses the model. Trim whole messages, keep call and result together, and never trim the instructions.

Interview questions

Q1What is a token, and why not just count words?

A token is the subword unit a model reads and writes, produced by a tokenizer such as BPE that merges frequent byte sequences into vocabulary entries. Because the vocabulary reflects frequency, common English words are one token while code, numbers and many languages split into several. Limits and prices are in tokens, so word counts underestimate exactly the inputs that are most expensive.

Q2Walk me through implementing a chat that never exceeds the context window.

I pick a budget at or below the window, reserve the output tokens that match max_tokens, and subtract the system prompt, tool definitions and the new message. Then I add history from newest to oldest, whole messages only and keeping tool calls with their results, until the budget is used. Dropped turns are folded into a running summary or stored for retrieval, important structured facts live in application state, and I count with the provider's tokenizer and check the stop reason on every response.

Q3What happens when the prompt plus max output exceeds the window?

Usually the API rejects the request with a context-length error; some providers instead shrink the available output, so the answer gets cut off. Either way it is a bug in the budget, not in the model. The fix is to trim history or retrieved context before sending, and to detect truncated output by its stop reason rather than by a JSON parse failure later.

Q4Why does a long conversation get slower and more expensive each turn?

The model is stateless, so every turn resends the full history, and input tokens per turn grow linearly with the turn number. Total tokens across the conversation therefore grow roughly quadratically, and time to first token rises because the model must process the whole prompt before answering. Trimming, summarising and prompt caching of the stable prefix all attack this.

Q5If a model accepts a million tokens, why use RAG?

Because capacity is not the same as cost, latency or accuracy. Sending a million tokens per question is expensive and slow, and models use long contexts unevenly, missing facts in the middle. Retrieval sends a few relevant passages, keeps prompts small and cacheable, and scales to corpora larger than any window. Long context is useful for one large document; RAG is for many documents and frequent queries.

Q6How does prompt caching work, and how do you structure prompts for it?

The provider stores the processed state of a prompt prefix and reuses it when a later request begins with exactly the same tokens, charging less and responding faster for that part. So I put the stable parts first, in a fixed order: system instructions, tool definitions, static reference text, and only then history, retrieved results and the new message. Anything dynamic near the top breaks the match for everything after it.

Q7How would you estimate the monthly cost of a new LLM feature?

Count tokens per call for a realistic request: fixed prompt, typical retrieved context, average history and expected output. Cost per call is input tokens times the input rate plus output tokens times the output rate, with cached input at its discounted rate. Multiply by calls per user per day, active users and days, add retries and multi-step agent calls, then confirm against real usage logs in a small rollout.

Key takeaways

Models read tokens, not words; English prose is about four characters per token, while code, IDs and other languages cost more. Count with the real tokenizer.
The context window holds input and output together, and it is the model's only memory: every call resends everything it needs.
Reserve the output budget first, then fit history and retrieved context into what is left; check the stop reason for truncation.
Long prompts raise cost and time to first token, and the middle of a long context is used least. Retrieve and place the relevant parts near the question.
Trim whole messages, keep tool calls with their results, and move durable facts into summaries or application state.
Put stable content first so prompt caching can reuse it; anything that changes per request goes last.

Sources

#Context

#Why it matters

#What a token is, and why counts surprise you

#The window is everything the model sees

Why every turn costs more than the last

#Longer is slower, pricier, and not always better

#Fitting a conversation into the budget

#Pitfalls

#Interview questions