The Production Incident

Our Order Management Backend is now running in production as release 1.6.0, with everything twelve modules built: Products, Inventory, Orders, payments, security, transactions, tests, observability, a container image and a CI pipeline. Until now, our work flowed in one direction: requirement, design, implementation, testing, shipping.

Now production sends information back to us. A Customer reports:

I tried to place one Order, the request appeared to fail, so I tried again. Now I have two Orders.

This begins Module 13, and its first lesson has an unusual goal. It is not "write a fix". It is:

Understand what actually happened.

The Incident Report

Support receives the Customer's message:

I submitted an Order. The app kept loading, and eventually I received an error. I tried again, and the second attempt succeeded. Later I noticed two Orders for the same items.

Support checks the Customer's Order history and finds two Orders for Customer customer-742, placed on 22 October 2026 shortly after 22:14 UTC:

We will call them O1 and O2. Both Orders are structurally valid, both belong to the same Customer, both contain valid OrderItems, and both consumed Inventory. This is not a page rendering one Order twice: there are two persisted Orders.

Facts Before Hypotheses

We can describe the known symptom carefully:

A Customer retry after an uncertain Create Order response resulted in two distinct Orders for what the Customer considered one logical purchase attempt.

The statement is deliberately cautious. We are not yet saying "frontend bug", "database bug", "network bug", "transaction bug" or "idempotency bug". Any of these may become a conclusion later, but at the beginning of an incident the most important skill is to keep observed facts apart from hypotheses.

The incident also deserves an engineering record, not only a chat thread. The repository layout from 3.01 already reserved docs/incidents/ for incident reviews, so this investigation gets docs/incidents/duplicate-order-after-retry.md. We build that document as the investigation progresses, and we do not write a root cause into it before we have investigated one.

The Corrective Backlog Item

Our backlog anticipated this kind of work in 1.08: a production issue may create "BUG-201 Prevent Duplicate Order Creation Under Retry". Our initial backlog ended at BACKEND-128, but a backlog represents current understanding, not eternal truth, and BUG-201 now joins it. Notice the wording. It does not say "add an idempotency table", "add a Redis lock" or "add a unique constraint". It states the behavior we need to protect, and the investigation will determine the design.

The First Rule During an Incident

Do not start coding immediately. An incident creates pressure, and someone will suggest a quick fix: just add a retry check, just compare recent Orders, just disable the button, just add a database constraint, just make the POST retry-safe somehow. Each sounds useful. Acting before understanding the failure, however, can hide evidence, introduce another bug, solve only one symptom or break legitimate behavior. The first engineering task is to establish facts; an Idempotency-Key header, a new table, Redis or a changed POST /api/v1/orders contract all wait until the root cause and the requirements are explicit.

What We Know and What We Must Determine

From the Customer, we know that one intended purchase appears to have created two Orders, that the first request did not appear successful and that the second one did. From PostgreSQL, we know that both Orders exist and that both consumed stock. That is enough to begin the investigation, not enough to claim a root cause. The investigation checklist moves us from "a duplicate Order happened" toward "how it happened":

1. Confirm that both Orders exist and that both transactions committed.
2. Compare their Customer and OrderItems.
3. Find the request IDs related to both Orders and reconstruct the request sequence.
4. Determine whether R2 was a retry, and what happened to R1's response.
5. Check the Inventory effects and the payment state.
6. Assess whether other Customers experienced similar behavior.
7. Determine whether a deployment, database health or network behavior contributed.
8. Identify the system condition that allowed one logical operation to create two Orders.

Start With Persistent Truth

Our transaction design established an important principle:

Persisted committed state is the source of truth.

So the investigation begins with PostgreSQL, not with the assumption "the client saw an error, therefore the first request failed". Customers describe what they saw, and their descriptions are evidence of the client's view only. A timeout means the client stopped waiting, not where the backend was: the request may never have arrived, may have failed in its transaction, or may have committed while the response was delayed or lost. Even a 500 does not prove a rollback, because a later step such as response serialization can fail after the commit. 7.10 already named this boundary: a transaction answers whether the database work committed atomically, not whether the Customer received the response.

A Suspected Risk, Not Yet a Root Cause

Be careful with hindsight here. In 7.10 we explicitly recognized that repeating a Create Order request after an uncertain network result could create duplicate Orders, and we chose not to invent an idempotency contract before a concrete requirement existed. That was intentional scope control, and the incident looks like that risk: our Create Order contract carries Product IDs and quantities but no request-level operation identity, so a retry would arrive as a new request and get a new OrderId.

That is a hypothesis, not a finding. We record it as a suspected risk and let the evidence confirm or refute it: 13.02 collects the evidence, and 13.03 reconstructs the timeline and states the root cause. Module 11's structured logs, request IDs, metrics and health checks were not decorative production features; they are now our evidence.

Reactions That Make Things Worse

An automatic server-side retry of Create Order. Create Order generates a new OrderId and consumes Inventory, so blindly rerunning it produces exactly the duplication we are investigating. Retry safety requires an operation identity and clear semantics, not "try again".

Deleting one Order. Our database deliberately protects historical state, we have no general Order deletion capability, and deleting production records destroys evidence. First establish which Order is valid, what happened downstream, whether payment happened and whether Inventory needs correction.

Fixing Inventory by hand. Changing available_quantity manually before understanding the Order state can create new inconsistency: restoring stock for O2 while O2 later becomes the legitimately paid Order leaves Inventory wrong. Operational remediation follows business truth, and investigation comes first.

Scoping the Impact

One Customer report does not tell us whether this is one incident or systematic production behavior. Impact analysis asks: how many duplicate-looking Orders exist? Did this start recently, with release 1.6.0 or earlier? Is it associated with timeouts, one client type or one application version? Are any duplicate Orders PAID? How much Inventory was consumed, and did any Product become unavailable incorrectly?

A query for "same Customer, same items, close timestamps" can find candidates; it cannot label them duplicates, because legitimate repeated purchases look identical. Candidate analysis needs more evidence: logs, the request timeline, support reports and request metadata.

Labeled Unknowns

Every statement in the investigation must later be tied to evidence, and the incident document sorts its statements accordingly:

• Confirmed: two Orders exist, they belong to the same Customer, and both consumed Inventory.
• Reported: the Customer did not receive the first successful response.
• Suspected: the second request was a retry of the same logical operation.
• Unknown: the exact reason the first response did not reach the Customer.

"Why did R1's response fail?" may remain unknown. Do not fill gaps with confidence you do not have.

Severity, Remediation and the Permanent Fix

This course has defined no organizational severity framework, so we do not invent SEV-1, P0 or Critical-7 without a real company convention. We can still describe potential impact: Inventory reduced twice, Customer confusion, support workload, and, if both Orders are later paid, a Customer charged twice. Do not claim any of these occurred; the investigation must assess them, starting with the status and payment attempts of O1 and O2.

We may need two tracks. Immediate remediation inspects the affected Customers' Orders, determines the valid business state and corrects it safely, within existing rules such as "an unresolved payment attempt blocks cancellation". Permanent corrective work prevents an equivalent retry from creating duplicate Orders. Repairing existing data and preventing future incidents are different tasks.

Do we need an emergency code change? Not necessarily. First determine frequency, impact, reproducibility and a safe mitigation. If incidents are frequent and severe, urgent mitigation may be appropriate; if evidence is incomplete, investigation is safer than rapidly shipping the wrong fix. Our scenario defines no specific operational environment, so we invent no support or feature-restriction procedures; this module focuses on understanding, designing, implementing and testing the corrective backend behavior.

The Initial Incident Record

At the end of this lesson, docs/incidents/duplicate-order-after-retry.md begins like this:

# Incident: Duplicate Order Created After Retry

Status: Under investigation

## Impact
At least one Customer observed two persisted Orders after retrying an Order
creation request whose first response appeared unsuccessful.

## Known
- Both Orders exist in PostgreSQL, with distinct UUIDs (6a74b0a7-…, 24b1a51f-…).
- Both belong to customer-742 and contain equivalent items.
- Both committed Inventory effects.
- The Customer reports retrying after an uncertain first response.

## Unknown
- The exact reason the first response was not received.
- Whether similar incidents affected other Customers.
- Whether either Order entered payment processing.
- The exact system timeline across both requests.

## Suspected Risk
Create Order has no request-level identity that lets the backend recognize
a retry of the same logical operation.

## Next Action
Reconstruct the incident from persisted state, structured logs, request IDs,
metrics and runtime evidence.

The weakness appears under "Suspected Risk", not yet under "Root Cause". That label comes after the investigation.

Practice Exercise 1: Sort the Statements

Solutions: lesson 13.11.

The incident chat collects these statements in its first hour. Label each one confirmed, reported, suspected or unknown, and rewrite any statement that claims more than its evidence:

1. "The first request failed."
2. "Both Orders are UNPAID."
3. "The app kept loading and then showed an error."
4. "The app retried automatically."
5. "Release 1.6.0 introduced the problem."
6. "O1 consumed Inventory."

Practice Exercise 2: Two Orders From One Body

Start release 1.6.0, the module 12 project in code/module-12, which has no idempotency: docker compose up -d --wait, then ./gradlew bootRun --args='--spring.profiles.active=local'. With token() from 8.05, let the administrator create a Monitor Stand at 4500 cents and a Wireless Mouse at 2990 cents and stock ten of each. Then Ana sends the same Create Order body, one stand and two mice, twice in a row.

Predict the two statuses and whether the Location headers match. Then open psql with docker compose exec postgres psql -U order_management -d order_management and list Ana's Orders (customer_id = '6f1c2a52-3b0e-4d6b-9a41-2c7e5d8f0a11'), the two Inventory rows and the payment attempts of both Orders. Write the "Known" section of an incident record for your two Orders. Nothing timed out here, so this is not yet the incident's reproduction (that comes in 13.03); it only tests whether the backend can tell a repeated body from a new Order.

Practice Exercise 3: Judge the Quick Fixes

Four suggestions arrive while the investigation is still open. For each, say what it could break and which fact you would need first:

1. "Let the app retry Create Order by itself after a timeout, so the Customer never sees the error."
2. "Cancel O2 now; it is the duplicate."
3. "Reject a Create Order whose items match the Customer's previous Order."
4. "Raise the client timeout from 10 to 60 seconds."

Knowledge Check

Question 1

Why does the investigation start in PostgreSQL rather than with the Customer's description of what happened?

Question 2

The client received a 500 for a Create Order request. Can you conclude that no Order was created?

Question 3

Why is BUG-201 worded as a behavior ("Prevent Duplicate Order Creation Under Retry") rather than as "Add an idempotency table"?

Question 4

A query finds two Orders by one Customer with identical items, a minute apart. Why is that not yet a duplicate?

Question 5

Why keep immediate remediation and the permanent fix as separate tracks?

Knowledge Check Answers

Answer 1

Committed state is the source of truth for whether an Order exists. The Customer's description is evidence of what the client saw, and a client can see an error for a request that committed.

Answer 2

No. A failure after the commit, such as in response serialization, returns a 500 while the Order and its Inventory effect are durable; only the database answers whether it exists.

Answer 3

The investigation has not yet found the cause. A behavior states what must be protected and leaves the design to the evidence, while a named mechanism would commit to a fix before anyone knows what failed.

Answer 4

Legitimate repeated purchases look exactly the same. Similarity makes the pair a candidate; logs, the request timeline or a support report decide whether it was one intended operation.

Answer 5

They answer different questions. Remediation corrects the existing Orders according to their business state, and the permanent fix stops future retries from creating new ones; a hurried mix of both risks doing either wrongly.

Engineering Principle

During an incident, establish what happened before deciding how to fix it. Production symptoms, hypotheses and confirmed facts are not the same thing.

Summary

• A Customer reports two Orders after retrying an uncertain Create Order request; PostgreSQL holds two distinct, individually valid Orders that both consumed Inventory.
• The investigation starts with facts, not fixes, and keeps confirmed facts, reports, hypotheses and unknowns apart; committed database state is the source of truth.
• A client-visible failure (a timeout or even a 500) does not prove the server failed.
• 7.10 deliberately postponed Create Order idempotency; the missing operation identity is a suspected risk, tracked as BUG-201, until the evidence confirms it.
• Automatic retries, deleting Orders or editing Inventory by hand make an incident worse; remediation follows business state.
• Candidate duplicates stay candidates, impact (including payment) must be verified, and remediation is separate from the permanent fix.

In the next lesson, Investigating with Production Signals, we take the incident from "the Customer says they got two Orders" to evidence: PostgreSQL state, created_at values, the ECS logs with requestId, orderId and customerId, payment attempts, HTTP metrics and health, while keeping evidence, correlation, inference and root cause apart.