Checking lesson access...
The Whole System in One Hour
One model call, one tool and one trace, end to end on the starter repository.
You are viewing a free preview lesson.
Situation
You have just become the owner of the Order Support Assistant, which answers an online shop's customers about their orders. Today it does one thing: a customer asks where an order is, a language model asks for an order lookup, the application performs it, and the model writes the answer. Before the foundation lessons explain each piece, you run the whole path once and read its trace.
Concept
The assistant has four parts, and the course keeps them apart from the first line of code:
- The application is ordinary Java. It holds the customer's identity, the order data and every rule about who may see what.
- The model (
gemma4:e2b, running locally in Ollama) reads text and proposes an answer or a tool call: a request to run one named operation with arguments. - The tool is application code the model may ask for: here
get_order_status, which reads one order. - The trace records each model call with its tokens, each tool call with its arguments and outcome, and how long each took.
The split carries the principle the course rests on: the model proposes, the application decides. The model can ask for any order number; whether the customer may see it is decided in the tool's Java code, with the customer id the application already holds, never one the model supplies. Model APIs and frameworks will change; that split will not.
Mechanism
One question takes at most two model calls and one round of tool calls:
Drawing diagram…
Each call to the model is a model.call span and the tool run a tool.call span, all under one assistant.request span. If the model answers without a tool, there is no second call.
Decision Point
The shop has three order operations: read an order's status, cancel an order, and change its delivery address. Which ones should the model be able to call today?
Worked Decision
Options: (a) all three, so the assistant is useful at once; (b) only the status lookup; (c) none, answering from a database query the application runs before calling the model.
We choose (b). A status lookup is read-only: a wrong call gives a wrong or empty answer, which is recoverable. Cancelling is a side effect: a wrong call loses a sale, and a retry could cancel twice; it needs validation, idempotency and perhaps a person's approval first (Modules 3 and 6). Option (c) is cheaper and fully deterministic, and right if every question names an order number; Module 2 asks that question properly. Rejected: (a), because it gives the assistant authority we cannot yet limit.
Your Decision
Apply the same reasoning to "change the delivery address". Should the model get that tool now, later, or never? Name what would have to be true first.
Implementation
You need Java 21, Docker and Ollama. Pull the model once (about 4.6 GB) and start the local trace viewer, Jaeger, from the project folder:
ollama pull gemma4:e2b
docker compose up -d
The Project
The starter project is a plain Java application built with Gradle (Spring Boot arrives in Module 6); the wrapper's properties pin the Gradle version:
<!-- file: settings.gradle -->rootProject.name = 'order-support-assistant'
distributionUrl=https\://services.gradle.org/distributions/gradle-9.8.2-bin.zip
plugins {
id 'application'
}
group = 'com.tarkev'
version = '0.0.1-SNAPSHOT'
java {
toolchain {
languageVersion = JavaLanguageVersion.of(21)
}
}
repositories {
mavenCentral()
}
dependencies {
implementation platform('io.opentelemetry:opentelemetry-bom:1.67.0')
implementation 'io.opentelemetry:opentelemetry-sdk'
implementation 'io.opentelemetry:opentelemetry-exporter-otlp'
implementation 'tools.jackson.core:jackson-databind:3.2.4'
testImplementation platform('org.junit:junit-bom:6.1.3')
testImplementation 'org.junit.jupiter:junit-jupiter'
testImplementation 'io.opentelemetry:opentelemetry-sdk-testing'
testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}
application {
mainClass = providers.gradleProperty('main').orElse('com.tarkev.ordersupport.Main')
}
tasks.named('test') {
useJUnitPlatform()
}
services:
jaeger:
image: jaegertracing/jaeger:2.22.0
ports:
- "16686:16686"
- "4318:4318"
Jaeger receives traces on port 4318 and shows them at http://localhost:16686. ./gradlew run starts Main; -Pmain=<class> runs another class, which later lessons use for experiments.
Orders and the Tool
The order store is an in-memory stand-in for the shop's order service, with two customers:
package com.tarkev.ordersupport.orders;
import java.time.LocalDate;
public record Order(String id, String customerId, String status, LocalDate expectedDelivery) {
}
package com.tarkev.ordersupport.orders;
import java.time.LocalDate;
import java.util.Map;
import java.util.Optional;
public final class InMemoryOrders {
private final Map<String, Order> orders = Map.of(
"1042", new Order("1042", "C-100", "SHIPPED", LocalDate.of(2026, 10, 27)),
"1043", new Order("1043", "C-100", "PROCESSING", LocalDate.of(2026, 11, 3)),
"2001", new Order("2001", "C-200", "DELIVERED", LocalDate.of(2026, 10, 20)));
public Optional<Order> find(String orderId) {
return Optional.ofNullable(orders.get(orderId));
}
}
A tool has two halves: a specification the model reads (name, description, a JSON schema for the arguments) and the code the application runs. The customer id comes from the caller, not from the model's arguments, and an order that belongs to someone else gets the same reply as one that does not exist, so the model cannot learn that it exists:
package com.tarkev.ordersupport.tools;
import java.util.Map;
public record ToolSpec(String name, String description, Map<String, Object> parameters) {
}
package com.tarkev.ordersupport.tools;
import com.tarkev.ordersupport.orders.InMemoryOrders;
import java.util.List;
import java.util.Map;
public final class OrderStatusTool {
public static final ToolSpec SPEC = new ToolSpec("get_order_status",
"Look up the status and expected delivery date of one of the customer's orders.",
Map.of("type", "object",
"properties", Map.of("orderId", Map.of("type", "string")),
"required", List.of("orderId")));
public record Result(String outcome, String text) {
}
private final InMemoryOrders orders;
public OrderStatusTool(InMemoryOrders orders) {
this.orders = orders;
}
public Result execute(String customerId, Map<String, Object> arguments) {
String orderId = String.valueOf(arguments.get("orderId"));
var order = orders.find(orderId);
if (order.isEmpty() || !order.get().customerId().equals(customerId)) {
String outcome = order.isEmpty() ? "not_found" : "not_authorized";
return new Result(outcome, "No order " + orderId + " was found for this customer.");
}
return new Result("found", "Order %s: status %s, expected delivery %s."
.formatted(orderId, order.get().status(), order.get().expectedDelivery()));
}
}
The Model Client
The application reaches the model through a small interface, so tests can script it:
package com.tarkev.ordersupport.model;
import java.util.Map;
public record ToolCall(String name, Map<String, Object> arguments) {
}
package com.tarkev.ordersupport.model;
import java.util.List;
public record Message(String role, String content, List<ToolCall> toolCalls, String toolName) {
public static Message system(String text) {
return new Message("system", text, List.of(), null);
}
public static Message user(String text) {
return new Message("user", text, List.of(), null);
}
public static Message toolResult(String toolName, String text) {
return new Message("tool", text, List.of(), toolName);
}
}
package com.tarkev.ordersupport.model;
import com.tarkev.ordersupport.tools.ToolSpec;
import java.util.List;
public interface ModelClient {
record Reply(Message message, String model, int promptTokens, int outputTokens) {
}
Reply chat(List<Message> messages, List<ToolSpec> tools);
}
The Ollama implementation sends one HTTP request to /api/chat. "think": false turns off the model's hidden reasoning, which would add tokens to every call, and temperature 0 makes runs easier to compare. Module 2 takes the request apart:
package com.tarkev.ordersupport.model;
import com.tarkev.ordersupport.tools.ToolSpec;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import tools.jackson.core.type.TypeReference;
import tools.jackson.databind.JsonNode;
import tools.jackson.databind.json.JsonMapper;
public final class OllamaModelClient implements ModelClient {
private static final TypeReference<Map<String, Object>> ARGUMENTS = new TypeReference<>() {
};
private final HttpClient http = HttpClient.newHttpClient();
private final JsonMapper json = JsonMapper.builder().build();
private final URI chatUri;
private final String model;
public OllamaModelClient(String baseUrl, String model) {
this.chatUri = URI.create(baseUrl + "/api/chat");
this.model = model;
}
@Override
public Reply chat(List<Message> messages, List<ToolSpec> tools) {
var body = Map.of("model", model, "stream", false, "think", false,
"options", Map.of("temperature", 0),
"messages", messages.stream().map(OllamaModelClient::toOllama).toList(),
"tools", tools.stream().map(tool -> Map.of("type", "function", "function", tool)).toList());
var request = HttpRequest.newBuilder(chatUri)
.header("Content-Type", "application/json")
.timeout(Duration.ofSeconds(120))
.POST(HttpRequest.BodyPublishers.ofString(json.writeValueAsString(body)))
.build();
JsonNode reply = json.readTree(send(request));
JsonNode message = reply.path("message");
var calls = new ArrayList<ToolCall>();
for (JsonNode call : message.path("tool_calls")) {
JsonNode function = call.path("function");
calls.add(new ToolCall(function.path("name").asString(),
json.convertValue(function.path("arguments"), ARGUMENTS)));
}
return new Reply(new Message("assistant", message.path("content").asString(), calls, null),
model, reply.path("prompt_eval_count").asInt(), reply.path("eval_count").asInt());
}
private String send(HttpRequest request) {
try {
var response = http.send(request, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() != 200) {
throw new IOException("Ollama returned " + response.statusCode() + ": " + response.body());
}
return response.body();
} catch (IOException e) {
throw new UncheckedIOException("Model call to " + chatUri + " failed", e);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IllegalStateException("Interrupted while calling the model", e);
}
}
private static Map<String, Object> toOllama(Message message) {
var out = new HashMap<String, Object>(Map.of("role", message.role(), "content", message.content()));
if (message.toolName() != null) {
out.put("tool_name", message.toolName());
}
if (!message.toolCalls().isEmpty()) {
out.put("tool_calls", message.toolCalls().stream().map(call -> Map.of("function",
Map.of("name", call.name(), "arguments", call.arguments()))).toList());
}
return out;
}
}
The Assistant and Its Trace
The assistant puts the parts together. It sends the instructions, the question and the tool list; if the model asks for tools, it runs them and calls the model once more. Every step is a span with the attributes you will read in Jaeger:
package com.tarkev.ordersupport;
import com.tarkev.ordersupport.model.Message;
import com.tarkev.ordersupport.model.ModelClient;
import com.tarkev.ordersupport.model.ToolCall;
import com.tarkev.ordersupport.tools.OrderStatusTool;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import java.util.ArrayList;
import java.util.List;
public final class OrderSupportAssistant {
static final String INSTRUCTIONS = """
You answer one customer's questions about their orders in an online shop.
Use the get_order_status tool to look up an order. Never guess a status or a date.
If the question is not about an order's status, say you can only help with order status.
""";
public record Answer(String text, String traceId) {
}
private final ModelClient model;
private final OrderStatusTool orderStatus;
private final Tracer tracer;
public OrderSupportAssistant(ModelClient model, OrderStatusTool orderStatus, Tracer tracer) {
this.model = model;
this.orderStatus = orderStatus;
this.tracer = tracer;
}
public Answer answer(String customerId, String question) {
Span span = tracer.spanBuilder("assistant.request").setAttribute("customer.id", customerId).startSpan();
try (Scope ignored = span.makeCurrent()) {
var messages = new ArrayList<>(List.of(Message.system(INSTRUCTIONS), Message.user(question)));
ModelClient.Reply reply = callModel(messages);
if (!reply.message().toolCalls().isEmpty()) {
messages.add(reply.message());
for (ToolCall call : reply.message().toolCalls()) {
messages.add(Message.toolResult(call.name(), runTool(customerId, call)));
}
reply = callModel(messages);
}
return new Answer(reply.message().content(), span.getSpanContext().getTraceId());
} finally {
span.end();
}
}
private ModelClient.Reply callModel(List<Message> messages) {
Span span = tracer.spanBuilder("model.call").startSpan();
try (Scope ignored = span.makeCurrent()) {
ModelClient.Reply reply = model.chat(messages, List.of(OrderStatusTool.SPEC));
span.setAttribute("model.name", reply.model());
span.setAttribute("model.prompt_tokens", reply.promptTokens());
span.setAttribute("model.output_tokens", reply.outputTokens());
span.setAttribute("model.tool_calls", reply.message().toolCalls().size());
return reply;
} catch (RuntimeException e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR);
throw e;
} finally {
span.end();
}
}
private String runTool(String customerId, ToolCall call) {
Span span = tracer.spanBuilder("tool.call").setAttribute("tool.name", call.name())
.setAttribute("tool.arguments", call.arguments().toString()).startSpan();
try (Scope ignored = span.makeCurrent()) {
if (!call.name().equals(OrderStatusTool.SPEC.name())) {
span.setAttribute("tool.outcome", "unknown_tool");
return "There is no tool named " + call.name() + ".";
}
OrderStatusTool.Result result = orderStatus.execute(customerId, call.arguments());
span.setAttribute("tool.outcome", result.outcome());
return result.text();
} finally {
span.end();
}
}
}
Tracing is set up once, sending spans to Jaeger over OTLP. Main acts for customer C-100, as if that customer had logged in:
package com.tarkev.ordersupport;
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
public final class Tracing {
private Tracing() {
}
public static OpenTelemetrySdk start(String serviceName) {
var exporter = OtlpHttpSpanExporter.builder()
.setEndpoint(System.getenv().getOrDefault("OTLP_TRACES_ENDPOINT",
"http://localhost:4318/v1/traces"))
.build();
var tracerProvider = SdkTracerProvider.builder()
.setResource(Resource.getDefault().toBuilder().put("service.name", serviceName).build())
.addSpanProcessor(BatchSpanProcessor.builder(exporter).build())
.build();
return OpenTelemetrySdk.builder().setTracerProvider(tracerProvider).build();
}
}
package com.tarkev.ordersupport;
import com.tarkev.ordersupport.model.OllamaModelClient;
import com.tarkev.ordersupport.orders.InMemoryOrders;
import com.tarkev.ordersupport.tools.OrderStatusTool;
public final class Main {
public static void main(String[] args) {
try (var openTelemetry = Tracing.start("order-support-assistant")) {
var model = new OllamaModelClient(env("OLLAMA_BASE_URL", "http://localhost:11434"),
env("OLLAMA_MODEL", "gemma4:e2b"));
var assistant = new OrderSupportAssistant(model, new OrderStatusTool(new InMemoryOrders()),
openTelemetry.getTracer("order-support"));
var answer = assistant.answer("C-100", String.join(" ", args));
System.out.println(answer.text());
System.out.println("trace " + answer.traceId());
}
}
static String env(String name, String fallback) {
String value = System.getenv(name);
return value == null || value.isBlank() ? fallback : value;
}
}
Running It
./gradlew run --args="Where is my order 1042?"
Your order 1042 has been **SHIPPED** and the expected delivery date is **2026-10-27**.
trace 028c7bbd30419080399e3157a90120d9
Open http://localhost:16686, choose the service order-support-assistant and open the trace with that id. Jaeger draws it as a timeline; these are its spans and the attributes the code set:
assistant.request 7796 ms customer.id=C-100
model.call 7089 ms model.name=gemma4:e2b model.prompt_tokens=142 model.output_tokens=21 model.tool_calls=1
tool.call 1 ms tool.name=get_order_status tool.arguments={orderId=1042} tool.outcome=found
model.call 701 ms model.name=gemma4:e2b model.prompt_tokens=203 model.output_tokens=33 model.tool_calls=0
The first model call took seven seconds because Ollama was loading the model; once loaded, the same call takes about 0.6 seconds, so a first run is no measure of latency. The model asked for one tool call with orderId 1042, the tool found the order, and the second call, 61 prompt tokens longer because it carries the tool result, wrote the answer. The tool took one millisecond: the time and the tokens are in the model calls. The ** in the answer is Markdown, which the shop's chat window may not render: a question for Module 2.
The Seven Artifacts
Owning the system means keeping, beside the code, the documents a real team keeps. The starter project ships them as templates under docs/ and one evaluation case under evals/; each is started in the lesson named below and updated from then on:
| Artifact | File | Started in |
|---|---|---|
| Architecture doc | docs/architecture.md | 2.10 |
| ADR log | docs/adr/ | 2.10 |
| Failure-mode and threat register | docs/failure-register.md | 3.06 |
| Runbook | docs/runbook.md | Module 6 |
| Evaluation set | evals/cases/ | 3.08 |
| Cost and latency ledger | docs/cost-ledger.csv | 2.10 |
| Postmortems | docs/postmortems/ | 8.09 |
# Order Support Assistant: Architecture
## Components
## Model and application boundary
## State
## Data flows
## Capacity notes
# ADR NNNN: Title
Status: proposed | accepted | superseded by NNNN · Date:
## Context
## Options
## Decision
## Consequences
## Rejected alternative and why
# Failure-Mode and Threat Register
| Failure or threat | Detection (trace field, metric, log) | Limit | Recovery | Since |
| --- | --- | --- | --- | --- |
# Runbook
## How to tell the assistant is unhealthy
## What to check first
## What to roll back, and how
## Who approves what
# Postmortem: Title
Date: · Trace ids:
## What happened
## Timeline (from traces)
## Root cause (operational or behavioral)
## What we changed (code or configuration)
## Added to the eval set or runbook
date,request_type,model,model_calls,tool_calls,prompt_tokens,output_tokens,duration_ms,notes
An evaluation case states an input and what a correct run does. This one is today's question; Module 3 writes a dozen more and Module 11 runs them:
<!-- file: evals/cases/order-status-1042.json -->{
"id": "order-status-1042",
"customerId": "C-100",
"input": "Where is my order 1042?",
"expect": {
"toolCalls": [{ "name": "get_order_status", "arguments": { "orderId": "1042" } }],
"answerMentions": ["shipped", "2026-10-27"]
}
}
A Test That Needs No Model
The test scripts the model and records spans in memory, so it runs in milliseconds with the same result every time. It also checks the authority rule: customer C-100 learns nothing about order 2001, whatever the model asks for:
package com.tarkev.ordersupport;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import com.tarkev.ordersupport.model.Message;
import com.tarkev.ordersupport.model.ModelClient;
import com.tarkev.ordersupport.model.ToolCall;
import com.tarkev.ordersupport.orders.InMemoryOrders;
import com.tarkev.ordersupport.tools.OrderStatusTool;
import com.tarkev.ordersupport.tools.ToolSpec;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.data.SpanData;
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.Test;
class OrderSupportAssistantTest {
private final InMemorySpanExporter spans = InMemorySpanExporter.create();
private final ScriptedModel model = new ScriptedModel();
private final OrderSupportAssistant assistant = new OrderSupportAssistant(model,
new OrderStatusTool(new InMemoryOrders()),
SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spans)).build().get("test"));
@Test
void looksUpTheOrderThroughTheTool() {
model.replies.add(toolCall("1042"));
model.replies.add(text("Your order 1042 has shipped."));
var answer = assistant.answer("C-100", "Where is my order 1042?");
assertEquals("Your order 1042 has shipped.", answer.text());
assertEquals("Order 1042: status SHIPPED, expected delivery 2026-10-27.", model.lastToolResult());
assertEquals(List.of("model.call", "tool.call", "model.call", "assistant.request"),
spans.getFinishedSpanItems().stream().map(SpanData::getName).toList());
}
@Test
void neverShowsAnotherCustomersOrder() {
model.replies.add(toolCall("2001"));
model.replies.add(text("I could not find that order."));
assistant.answer("C-100", "Where is order 2001?");
assertFalse(model.lastToolResult().contains("DELIVERED"));
assertEquals("not_authorized", spans.getFinishedSpanItems().get(1).getAttributes()
.get(AttributeKey.stringKey("tool.outcome")));
}
private static ModelClient.Reply toolCall(String orderId) {
var call = new ToolCall("get_order_status", Map.of("orderId", orderId));
return new ModelClient.Reply(new Message("assistant", "", List.of(call), null), "scripted", 0, 0);
}
private static ModelClient.Reply text(String content) {
return new ModelClient.Reply(new Message("assistant", content, List.of(), null), "scripted", 0, 0);
}
static final class ScriptedModel implements ModelClient {
final ArrayDeque<Reply> replies = new ArrayDeque<>();
final List<List<Message>> received = new ArrayList<>();
@Override
public Reply chat(List<Message> messages, List<ToolSpec> tools) {
received.add(List.copyOf(messages));
return replies.removeFirst();
}
String lastToolResult() {
return received.getLast().getLast().content();
}
}
}
./gradlew test
What Can Fail, and How We Would Know
| Failure | Detection | Limit | Recovery |
|---|---|---|---|
| Ollama is not running or the model is not pulled | model.call span with status ERROR and a ConnectException (or HTTP 404) event; no answer printed | 120-second request timeout | Start Ollama, ollama pull gemma4:e2b; Module 6 adds a fallback answer |
| The model answers without calling the tool and invents a status | One model.call with model.tool_calls = 0 and no tool.call span, on a question that names an order | None yet: instructions only ask it not to guess | Module 2 validates output; Module 11 adds eval cases that catch it |
| The model asks for another customer's order | tool.call with tool.outcome = not_authorized | Enforced in OrderStatusTool, not in the prompt | Nothing to recover: the tool returned no data |
| The model asks for a tool that does not exist | tool.call with tool.outcome = unknown_tool | The application only runs tools it knows | The model gets an error text and answers once more |
| Jaeger is down | A warning from the OTLP exporter in the console; the answer still prints | Tracing never blocks an answer | docker compose up -d; the lost trace is gone |
Owning It
You now own this system, and owning it starts with being able to show what it did. Your first duty is a habit: every wrong answer you report comes with its trace id. The trace shows the model's tool calls, their arguments and outcomes, and each call's tokens, so nobody argues from memory about what the model "probably" did. From Module 6 this is the runbook's first rule.
Exercise
Model answers: lesson 1.02.
Design
Write your decision on the delivery-address tool in five or six sentences: the options, your choice, what must be true before the model gets it, and the alternative you rejected. Then add two failures that tool could bring to the table above, each with the span attribute that would reveal it.
Build
- Run
./gradlew testand then the assistant with three questions:"Where is my order 1042?","Where is order 2001?"and"Can I return a jacket?". - For each, find the trace in Jaeger and write down: how many
model.callspans it has, whether atool.callappears and with whichtool.outcome, and the prompt and output tokens of each call. - One of the three questions should not need the tool. Did the model call it anyway? What would you change, and in which part of the system: the instructions, the tool description or the Java code?
Check Yourself
Question 1
The model asks get_order_status for order 2001 on behalf of customer C-100. Why does the tool reply exactly as it would for an order that does not exist?
Question 2
A question that names an order produces a trace with one model.call and no tool.call. What has probably happened, and why can the current system not stop it?
Question 3
Why does the test use a scripted model instead of gemma4:e2b, and what does it still not tell you about the real system?
Think first
Coming soonCheckpoint questions that ask you to commit to an answer before the lesson shows it.