Tokens, Tokenization, and Context Windows

Situation

A customer pasted a whole email thread into the chat: their original question about order 7731, then six hundred lines of quoted replies. The assistant answered confidently with the wrong order number. Nothing failed: no exception, no error in the trace, a normal done_reason of stop. The model simply never saw the first line. To own a system built on a model you have to know what the model actually reads, and that is measured in tokens, inside a window you configure.

Concept

Tokens

A model does not read characters or words. Its input is split into tokens: pieces of text from a fixed vocabulary the model was trained with, often a whole common word, sometimes a fragment, a single digit or a punctuation mark. The split is done by the model's tokenizer, and each model family has its own, so the same text costs a different number of tokens on different models.

Ollama has no endpoint that only counts tokens, but every reply reports prompt_eval_count, the tokens the model read. Sending each text alone, subtracting the nine tokens an empty message costs, gives these counts for gemma4:e2b:

tokens  chars  text
    10     23  'Where is my order 1042?'
    10     29  'Wo ist meine Bestellung 1042?'
    11     25  'Where is my order 1042? 🙂'
     1      6  'refund'
     9     34  'Bestellungsstornierungsbestätigung'
     4      4  '1042'
    26     57  'Order 1042: status SHIPPED, expected delivery 2026-10-27.'
    12     29  'Hi, just checking in on 1043.'

Three things in this table matter for the assistant. Numbers are expensive: 1042 is four tokens, one per digit, so an order number, a date or an amount costs more than the word "refund". Long compound words split into many pieces. And the ratio of characters to tokens depends on the text: across the twenty messages of the 2.01 sample it is 3.8 characters per token (935 characters, 246 tokens), but a tool result full of digits and dates comes closer to two.

Why Tokens Are the Unit That Matters

Tokens are the unit of three things you own:

• Cost. Hosted providers price input and output tokens separately, per million. Running locally, a token costs compute time instead of money, but the ledger in 2.10 still converts tokens to a price, because that is the number a decision about models is made in.
• Latency. The model reads the prompt in one pass (fast) and writes the answer one token at a time (slow). In 1.01's trace, the second call read 203 tokens and wrote 33; the writing dominated its 701 milliseconds.
• Limits. Everything the model sees and writes in one call must fit in its context window.

The Context Window

The context window is the maximum number of tokens one call can hold: the instructions, the tool definitions, every message of the conversation, every tool result, and the tokens the model writes in reply. The model has no other memory. What does not fit in the window does not exist for that call.

There are two numbers, and confusing them causes the bug in the Situation. The model's trained context length is what it can handle at most: ollama show gemma4:e2b reports 131,072 tokens. The runtime context length is what the server actually allocates when it loads the model, and it is usually far smaller, because memory grows with it. Ollama chose 4,096 for this model on this laptop:

$ ollama ps
NAME          ID              SIZE      PROCESSOR    CONTEXT    UNTIL
gemma4:e2b    b37049369adf    237 MB    100% GPU     4096       4 minutes from now

The runtime length can be raised per request with the num_ctx option, at the price of memory and of a reload when it changes. A hosted provider makes the same distinction in its own terms: a model's advertised window, and the limits of your account and endpoint.

Mechanism

When a prompt is longer than the runtime window, Ollama does not refuse it. It keeps part of the prompt and discards the rest, and the reply looks normal. The email thread from the Situation was reproduced as one message: the order number in the first sentence, six hundred lines of filler, and the question at the end. Sent twice, once with the default window and once with num_ctx set to 16,384:

num_ctx default (4096)  prompt_eval_count 2051  done_reason stop  answer '465'
num_ctx 16384           prompt_eval_count 8929  done_reason stop  answer '7731'

The message was 8,929 tokens. With the default window the model read 2,051 of them, lost the first sentence, and answered 465, a number from the filler, with the same done_reason as the correct run. The only trace of the problem is that prompt_eval_count is smaller than the prompt, and you only notice that if you know how big the prompt was.

The application is the only part of the system that knows what it put into the prompt and why. So the application has to check the budget before the call, and decide what happens when it does not fit, rather than letting the server truncate.

Decision Point

Customers will paste long text. What should the assistant do with a message that does not fit?

Worked Decision

The options:

1. Raise num_ctx until everything fits. The model then reads the whole thread. Memory and latency grow with the window, the server reloads the model when the size changes, and there is still a limit, just a later one.
2. Truncate the message to fit, keeping the start or the end. Cheap and automatic, and exactly the silent failure in the Mechanism, now done on purpose: whichever end we cut may hold the question.
3. Set a window sized from real traffic, estimate every prompt before the call, and refuse what does not fit, with a fixed reply that hands the conversation to a person and a trace attribute that counts how often it happens.

We choose option 3, with an 8,192-token window. The 2.01 sample averages 12 tokens per message and the longest is under 25; with the instructions and the tool definition, 1.01's calls used 142 and 203 tokens. 8,192 leaves room for a long conversation and long tool results while keeping memory modest on a laptop. A message that still does not fit is rare, and a person reading it is a better outcome than an answer about the wrong order. Rejected: option 2, because it turns an occasional, visible refusal into an occasional, invisible wrong answer.

Your Decision

From Module 3, tools return data into the context. Suppose a customer with 300 past orders asks "which of my orders haven't arrived?", and a tool that lists orders returns all 300. The prompt no longer fits. Where should the limit live: in the tool, in the assistant, or in the model's instructions? Say what the limit is and what the customer sees.

Implementation

The code needs three things: an estimate of a prompt's tokens that errs on the high side, a check in the model client before any request is sent, and a fixed answer in the assistant when the check fails.

An Estimate That Errs High

Counting exactly needs a call to the model, which is the cost we are trying to avoid. A character-based estimate is enough if it is pessimistic: 3 characters per token, against the 3.8 measured on the sample, overestimates English text by about a quarter and stays close for digit-heavy tool results. Each message also carries a few tokens of formatting (the role markers the chat template adds), so the estimate adds a fixed amount per message:

package com.tarkev.ordersupport.model;

import com.tarkev.ordersupport.tools.ToolSpec;
import java.util.List;

public final class TokenEstimate {
    // Measured on the 2.01 sample with gemma4:e2b: 3.8 characters per token.
    // 3.0 errs high on purpose; re-measure when the model changes.
    static final double CHARS_PER_TOKEN = 3.0;
    static final int TOKENS_PER_MESSAGE = 8;

    private TokenEstimate() {
    }

    public static int of(List<Message> messages, List<ToolSpec> tools) {
        int chars = 0;
        for (Message message : messages) {
            chars += message.content().length();
        }
        for (ToolSpec tool : tools) {
            chars += tool.name().length() + tool.description().length() + tool.parameters().toString().length();
        }
        return (int) Math.ceil(chars / CHARS_PER_TOKEN) + TOKENS_PER_MESSAGE * messages.size();
    }
}

An overflow is a condition the caller must handle, not a crash, so it gets its own exception with the numbers in it:

package com.tarkev.ordersupport.model;

public final class ContextOverflow extends RuntimeException {
    private final int estimatedTokens;
    private final int contextLength;

    public ContextOverflow(int estimatedTokens, int contextLength) {
        super("Prompt of about " + estimatedTokens + " tokens does not fit a window of " + contextLength);
        this.estimatedTokens = estimatedTokens;
        this.contextLength = contextLength;
    }

    public int estimatedTokens() {
        return estimatedTokens;
    }

    public int contextLength() {
        return contextLength;
    }
}

The Client Checks Before It Sends

The Ollama client now takes the context length, sends it as num_ctx so the server allocates the window the check assumes, and refuses a prompt that would not fit with room left for the reply. The members below replace the constructor and chat of 1.01's client; the rest of the class is unchanged:

private static final int RESERVED_FOR_REPLY = 1024;
private final int contextLength;

public OllamaModelClient(String baseUrl, String model, int contextLength) {
    this.chatUri = URI.create(baseUrl + "/api/chat");
    this.model = model;
    this.contextLength = contextLength;
}

@Override
public Reply chat(List<Message> messages, List<ToolSpec> tools) {
    int estimate = TokenEstimate.of(messages, tools);
    if (estimate > contextLength - RESERVED_FOR_REPLY) {
        throw new ContextOverflow(estimate, contextLength);
    }
    var body = Map.of("model", model, "stream", false, "think", false,
            "options", Map.of("temperature", 0, "num_ctx", contextLength),
            "messages", messages.stream().map(OllamaModelClient::toOllama).toList(),
            "tools", tools.stream().map(tool -> Map.of("type", "function", "function", tool)).toList());
    var request = HttpRequest.newBuilder(chatUri)
            .header("Content-Type", "application/json")
            .timeout(Duration.ofSeconds(120))
            .POST(HttpRequest.BodyPublishers.ofString(json.writeValueAsString(body)))
            .build();
    JsonNode reply = json.readTree(send(request));
    JsonNode message = reply.path("message");
    var calls = new ArrayList<ToolCall>();
    for (JsonNode call : message.path("tool_calls")) {
        JsonNode function = call.path("function");
        calls.add(new ToolCall(function.path("name").asString(),
                json.convertValue(function.path("arguments"), ARGUMENTS)));
    }
    return new Reply(new Message("assistant", message.path("content").asString(), calls, null),
            model, reply.path("prompt_eval_count").asInt(), reply.path("eval_count").asInt());
}

The reply reserve of 1,024 tokens is generous for a two-sentence answer; it also covers the error of the estimate. Sending num_ctx on every request matters as much as the check: without it, the check would assume a window the server never allocated.

The Assistant Hands Over Instead of Guessing

answer catches the overflow and returns a fixed reply, and records the outcome on the request span so overflows can be counted. callModel adds the estimate next to the real count, so every trace shows how good the estimate is. Both methods replace their 1.01 versions:

import com.tarkev.ordersupport.model.ContextOverflow;
import com.tarkev.ordersupport.model.TokenEstimate;

static final String TOO_LONG = "That message is too long for me to read in one go. "
        + "A member of the support team will reply to you directly.";

public Answer answer(String customerId, String question) {
    Span span = tracer.spanBuilder("assistant.request").setAttribute("customer.id", customerId).startSpan();
    try (Scope ignored = span.makeCurrent()) {
        var messages = new ArrayList<>(List.of(Message.system(INSTRUCTIONS), Message.user(question)));
        ModelClient.Reply reply = callModel(messages);
        if (!reply.message().toolCalls().isEmpty()) {
            messages.add(reply.message());
            for (ToolCall call : reply.message().toolCalls()) {
                messages.add(Message.toolResult(call.name(), runTool(customerId, call)));
            }
            reply = callModel(messages);
        }
        span.setAttribute("assistant.outcome", "answered");
        return new Answer(reply.message().content(), span.getSpanContext().getTraceId());
    } catch (ContextOverflow e) {
        span.setAttribute("assistant.outcome", "context_overflow");
        return new Answer(TOO_LONG, span.getSpanContext().getTraceId());
    } finally {
        span.end();
    }
}

private ModelClient.Reply callModel(List<Message> messages) {
    Span span = tracer.spanBuilder("model.call").startSpan();
    try (Scope ignored = span.makeCurrent()) {
        span.setAttribute("model.prompt_tokens_estimate", TokenEstimate.of(messages, List.of(OrderStatusTool.SPEC)));
        ModelClient.Reply reply = model.chat(messages, List.of(OrderStatusTool.SPEC));
        span.setAttribute("model.name", reply.model());
        span.setAttribute("model.prompt_tokens", reply.promptTokens());
        span.setAttribute("model.output_tokens", reply.outputTokens());
        span.setAttribute("model.tool_calls", reply.message().toolCalls().size());
        return reply;
    } catch (RuntimeException e) {
        span.recordException(e);
        span.setStatus(StatusCode.ERROR);
        throw e;
    } finally {
        span.end();
    }
}

Main reads the window from the environment, so it can change without a rebuild:

public static void main(String[] args) {
    try (var openTelemetry = Tracing.start("order-support-assistant")) {
        var model = new OllamaModelClient(env("OLLAMA_BASE_URL", "http://localhost:11434"),
                env("OLLAMA_MODEL", "gemma4:e2b"), Integer.parseInt(env("OLLAMA_NUM_CTX", "8192")));
        var assistant = new OrderSupportAssistant(model, new OrderStatusTool(new InMemoryOrders()),
                openTelemetry.getTracer("order-support"));
        var answer = assistant.answer("C-100", String.join(" ", args));
        System.out.println(answer.text());
        System.out.println("trace " + answer.traceId());
    }
}

Testing the Limit Without a Model

The check runs before any request, so a test can point the client at a port where nothing listens: if the client tried to send, the test would fail with a connection error instead of ContextOverflow:

package com.tarkev.ordersupport.model;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

import com.tarkev.ordersupport.tools.OrderStatusTool;
import java.util.List;
import org.junit.jupiter.api.Test;

class ContextLimitTest {
    private final OllamaModelClient client = new OllamaModelClient("http://localhost:1", "gemma4:e2b", 8192);

    @Test
    void refusesAPromptThatDoesNotFitBeforeSendingIt() {
        var thread = "Line of a quoted email thread. ".repeat(1500);
        var overflow = assertThrows(ContextOverflow.class,
                () -> client.chat(List.of(Message.user(thread)), List.of(OrderStatusTool.SPEC)));
        assertEquals(8192, overflow.contextLength());
        assertTrue(overflow.estimatedTokens() > 8192 - 1024);
    }

    @Test
    void estimatesHighForTheSampleMessages() {
        // 'Where is my order 1042?' is 10 tokens for gemma4:e2b (measured in 2.02).
        assertTrue(TokenEstimate.of(List.of(Message.user("Where is my order 1042?")), List.of()) >= 10);
    }
}

The 1.01 test builds the assistant with a scripted model, so it needs no change. Run the tests, then the pasted thread from the Situation and the normal question:

./gradlew test
./gradlew run --args="$(python3 -c 'print("My order is 7731. " + "Quoted reply line. " * 2500 + "What is my order number?")')"
./gradlew run --args="Where is my order 1042?"

That message is too long for me to read in one go. A member of the support team will reply to you directly.
trace 9667a1faea3c63bf3d4f6222a10c02ed
Your order 1042 has been **SHIPPED** and the expected delivery date is **2026-10-27**.
trace 2be5c0e95065d27aab4bfaf4a083f83d

The two traces, with the estimate beside the real count:

assistant.request       3 ms  assistant.outcome=context_overflow  customer.id=C-100
  model.call            2 ms  model.prompt_tokens_estimate=16000  error=true
                              exception: Prompt of about 16000 tokens does not fit a window of 8192

assistant.request    7784 ms  assistant.outcome=answered  customer.id=C-100
  model.call         7064 ms  model.prompt_tokens_estimate=160  model.prompt_tokens=142  model.output_tokens=21  model.tool_calls=1
  tool.call             1 ms  tool.name=get_order_status  tool.arguments={orderId=1042}  tool.outcome=found
  model.call          717 ms  model.prompt_tokens_estimate=195  model.prompt_tokens=203  model.output_tokens=33  model.tool_calls=0

The pasted thread never reached the model: the request took three milliseconds and the customer was told a person will reply. The normal question still works, and its first call took seven seconds again, because Ollama reloaded the model when the requested window changed from 4,096 to 8,192.

The estimates are worth reading closely. The first call was estimated at 160 tokens and used 142, as intended. The second was estimated at 195 and used 203: below the real count. TokenEstimate counts each message's content, but the assistant's tool-call message has empty content and carries its call (get_order_status with {orderId=1042}) in tool_calls, which the estimate ignores, and the tool message adds template tokens of its own. Eight tokens are absorbed by the reply reserve, so nothing breaks. But it is exactly the third row of the table below, found in the first real trace, and the exercise asks you to fix it.

What Can Fail, and How We Would Know

Owning It

You now own the context budget: the window size in OLLAMA_NUM_CTX, the reply reserve, and the characters-per-token constant behind the estimate. They are three numbers in code, and each one was chosen from a measurement. The duty that comes with them: when the model changes, re-measure the sample's tokens before you trust the estimate again, and watch the context_overflow count after any change to instructions or tools, since both make every prompt longer.

Exercise

Model answers: lesson 2.11.

Design

Write your answer to Your Decision for the 300-order tool result: where the limit lives, what it is (a number with its reason), what the customer sees, and which trace attribute would tell you the limit is hit too often.

Build

1. Measure your own setup: run ollama ps with the assistant's model loaded and note the context. Then run the assistant once with OLLAMA_NUM_CTX=2048 and the question "Where is my order 1042?". Does it still answer? Compare model.prompt_tokens with model.prompt_tokens_estimate in Jaeger.
2. Find the smallest window that still answers the question with the tool call. What uses most of the tokens: the instructions, the tool definition, or the question?
3. Make the estimate count tool calls: add each call's name and arguments (call.arguments().toString()) to the characters in TokenEstimate.of. Rerun "Where is my order 1042?" and check that the second call's estimate is now above its real count.
4. Add a test to ContextLimitTest that proves a message just under the limit is sent (point the client at the unreachable port again and expect the connection error, not ContextOverflow).

Check Yourself

Question 1

ollama show says the model handles 131,072 tokens. Why did a 9,000-token message still lose its first sentence?

Question 2

The estimate assumes 3 characters per token, but the sample measured 3.8. Why is the less accurate number the better one here?

Question 3

A colleague proposes cutting long messages to their last 2,000 characters instead of refusing them, "because the question is usually at the end". What would you need to know before agreeing, and what new failure would you add to the table?