Handling Production Incident
The Production Incident
Our Order Management Backend is now running as a production-style system.
We have already built:
Product management
Inventory management
Order creation and cancellation
Payment integration
authentication and authorization
PostgreSQL transactions
automated testing
structured logging
request correlation
health checks
metrics
container packaging
CI
Until now, most of our engineering work followed this direction:
requirement
↓
design
↓
implementation
↓
testing
↓
shipping
Now something changes.
Production sends information back to us.
A Customer reports:
I tried to place one Order, the request appeared to fail, so I tried again. Now I have two Orders.
This begins:
Module 13 — Investigating a Production Incident
Our first goal is not:
write a fix
Our first goal is:
Understand what actually happened.
The Incident Report
Support receives a Customer report.
The Customer says:
I submitted an Order.
The application kept loading.
Eventually I received an error.
I tried again.
The second attempt succeeded.
Later I noticed two Orders
for the same items.
Support checks the Customer's Order history.
They find two Orders.
Conceptually:
Order A
customerId = customer-742
status = UNPAID
items:
Product P1 × 2
Product P2 × 1
and shortly afterwards:
Order B
customerId = customer-742
status = UNPAID
items:
Product P1 × 2
Product P2 × 1
The IDs are different.
For example:
Order A
36a25516-2af2-4644-b55d-d0cabf389c9e
Order B
b29c51cc-f75a-46ae-a74c-b4c342547a20
Both Orders are structurally valid.
Both belong to the same Customer.
Both contain valid OrderItems.
Both consumed Inventory.
This is not merely:
duplicate UI rendering
There are actually two persisted Orders.
Initial Incident Statement
We can describe the known production symptom as:
A Customer retry after an uncertain Create Order response resulted in two distinct Orders for what the Customer considered one logical purchase attempt.
That statement is intentionally careful.
We are not yet saying:
frontend bug
or:
database bug
or:
network bug
or:
transaction bug
or:
idempotency bug
Those may become conclusions later.
At the beginning of an incident, distinguish:
observed facts
from:
hypotheses
Create an Incident, Not Just a Slack Thread
Earlier in the course we reserved:
docs/incidents/
for production incident reviews.
This incident should become part of the engineering record.
Conceptually:
docs/
└── incidents/
└── duplicate-order-after-retry.md
We will build that document as the investigation progresses.
Do not write the final root cause before investigating it.
Create the Corrective Backlog Item
The original course plan already anticipated a bug such as:
BUG-201
Prevent duplicate order creation
under identified retry scenario
Notice the wording.
It does not say:
Add an idempotency table.
It does not say:
Add Redis lock.
It does not say:
Add UNIQUE(customer_id, product_id).
It states the behavior we need to protect.
The investigation will determine the appropriate design.
The First Rule During an Incident
Do not start coding immediately.
A production incident creates pressure.
Someone may say:
Just add a retry check.
Just compare recent Orders.
Just disable the button.
Just add a database constraint.
Just make the POST retry-safe somehow.
Those may sound useful.
But acting before understanding the failure can:
hide evidence
introduce another bug
solve only one symptom
break legitimate behavior
The first engineering task is:
establish facts
What Do We Know?
From the Customer report:
one intended purchase
appears to have created:
two Orders
From PostgreSQL:
both Orders exist
From Inventory:
both Orders consumed stock
From the second request:
the Customer eventually received success
From the Customer:
the first request did not appear successful
That is enough to begin investigation.
It is not yet enough to claim exact root cause.
What Do We Need to Determine?
Important questions:
Did the first request reach the backend?
Did CreateOrderUseCase execute?
Did the first transaction commit?
Was a 201 response generated?
Did the client receive it?
Was the second request a retry?
Were the request payloads equivalent?
Did both requests have different request IDs?
Did both requests generate new OrderIds?
Was Inventory consumed twice?
Were there any database rollbacks?
Was there an application restart?
Was there an HTTP/network interruption?
Are other Customers affected?
These questions move us from:
duplicate Order happened
toward:
how duplicate Order happened
Start With Persistent Truth
Our earlier transaction design established an important principle:
Persisted committed state is the source of truth.
So begin with PostgreSQL.
Do not begin by assuming:
the client saw an error
therefore the first request failed
That conclusion is unsafe.
Client Failure Is Not Necessarily Backend Failure
Consider:
server successfully commits
and:
client reports timeout
These statements can both be true.
The HTTP connection and the database transaction are separate boundaries.
The client knows:
I did not receive a successful response.
It does not automatically know:
the backend transaction rolled back.
That distinction is central to this incident.
Revisit the Create Order Transaction
Our Create Order workflow is approximately:
validate request
↓
validate Products
↓
capture current priceCents
↓
consume Inventory
↓
construct Order
↓
persist Order + OrderItems
↓
COMMIT
All local database mutation is protected by the transaction.
The golden rule remains:
complete Order + Inventory mutation
or
no committed mutation
That protects database consistency.
What the Transaction Guarantees
Suppose the transaction commits.
Then:
Inventory consumption is committed
Order exists
OrderItems exist
Suppose it rolls back.
Then:
Inventory mutation disappears
Order does not exist
OrderItems do not exist
This prevents:
Inventory consumed
but no Order
or:
partial OrderItems
That is strong transactional correctness.
What the Transaction Does Not Guarantee
It does not guarantee:
the Customer received the HTTP response
Consider:
BEGIN
consume Inventory
persist Order
COMMIT
Then:
server tries to return 201
Between those events:
network connection may disappear
application process may terminate
proxy may time out
client may disconnect
response may be lost
PostgreSQL cannot roll back a transaction that already committed merely because the HTTP response later became uncertain.
The Dangerous Window
The failure window looks like:
Create Order request
↓
transaction executes
↓
COMMIT succeeds
↓
Order is now durable
↓
X
response does not reach Customer
From the backend's database perspective:
SUCCESS
From the Customer's perspective:
UNKNOWN
This is a very different problem from:
transaction failed
The Customer Retries
The Customer sees:
timeout
or another uncertain outcome.
Retrying seems reasonable.
The Customer sends another:
POST /api/v1/orders
with the same intended items.
Our current Create Order contract contains:
Product IDs
quantities
but no request-level operation identity.
The backend therefore sees:
new HTTP request
and executes:
new Create Order operation
A New Request Generates a New OrderId
Our canonical ID strategy is:
application-generated UUID
Each Create Order execution generates a new:
OrderId
So:
Request 1
→ OrderId O1
and:
Request 2
→ OrderId O2
Nothing in the current contract tells the backend:
these two requests represent
the same logical Customer operation
Both Requests Can Be Individually Valid
This is the subtle part.
Request 1 may be valid.
Request 2 may also be valid.
For each execution independently:
Customer authenticated
Products exist
Products are active
Inventory is sufficient
quantities are valid
Order Domain invariants hold
database transaction succeeds
So the backend can create:
two perfectly valid Orders
while still violating the Customer's intended behavior.
Database Integrity Does Not Detect This
Our database protects:
valid UUID primary keys
valid foreign keys
valid Order status
positive quantities
non-negative prices
unique Product within one Order
But these two records:
Order O1
Customer C1
Product P1 × 2
and:
Order O2
Customer C1
Product P1 × 2
do not violate any relational constraint.
Customers are allowed to place the same Order twice legitimately.
Therefore we cannot simply declare:
same Customer
+
same Products
=
duplicate
That would be incorrect.
Legitimate Duplicate-Looking Orders Exist
Imagine a Customer intentionally buys:
2 keyboards
at 10:00.
Then five minutes later buys:
2 more keyboards
Those are two legitimate Orders.
Or:
same Products
same quantities
same Customer
could represent genuine repeated purchases.
So this constraint would be wrong:
UNIQUE(
customer_id,
items
)
Even if such a schema were easy to express, the business meaning would be incorrect.
Time-Based Deduplication Is Also Dangerous
Another tempting fix:
If same Customer submits
same items within 30 seconds,
reuse previous Order.
What happens when the Customer genuinely wants:
two separate purchases
within 20 seconds?
The backend would silently discard legitimate intent.
Time similarity is evidence.
It is not a reliable operation identity.
UI Button Disabling Does Not Solve It
The frontend could disable:
Place Order
after the first click.
That is useful UX.
It does not solve backend correctness.
Why?
Retries may originate from:
browser
mobile application
proxy
SDK
network retry logic
manual API call
And the incident does not require a double-click.
The first request may genuinely become ambiguous.
The backend still needs a safe semantic answer for retries.
Request ID Does Not Solve It
We introduced:
X-Request-Id
for observability.
Could we deduplicate on:
requestId
?
Not automatically.
Request ID means:
this HTTP execution
If a client retries:
Request A
requestId = R1
Request B
requestId = R2
those are correctly two request executions.
Operational correlation is not business idempotency.
OrderId Does Not Solve It Either
The server currently generates the OrderId.
The client does not know:
OrderId O1
until it receives the successful response.
If that response is lost, the client cannot necessarily retry with:
OrderId O1
because it never learned it.
So current server-generated Order identity does not identify the logical client operation.
Payment Idempotency Does Not Solve Create Order Idempotency
We already studied idempotency carefully around Payment.
But Payment has its own:
PaymentAttempt
coordination.
That does not automatically make:
POST /api/v1/orders
idempotent.
Different operations require different identities and semantics.
Do not assume:
we solved idempotency once
therefore all POST requests are safe to retry
HTTP POST Is Not Automatically Retry-Safe
Our Create Order endpoint:
POST /api/v1/orders
creates new state.
If executed twice:
execution 1
→ O1
execution 2
→ O2
That is normal POST behavior unless we deliberately define additional idempotency semantics.
The incident is exposing a requirement we did not previously implement.
Was the Original Design Wrong?
Be careful with hindsight.
Earlier we explicitly recognized:
request-level idempotency
as a potential production concern.
We chose not to invent an idempotency contract before a concrete requirement existed.
That was intentional scope control.
Now production has given us:
concrete failure
concrete impact
concrete requirement
The backlog should evolve.
This is how long-lived systems work.
Production Changes Requirements
Before the incident, requirement was effectively:
Customer can create an Order.
Now we have learned an additional behavioral requirement:
Retrying the same logical Create Order operation after an uncertain response must not create another Order.
That is new product/system behavior.
It deserves design.
But We Are Not Designing It Yet
This lesson is about the incident.
Do not jump directly to:
Idempotency-Key header
or:
new table
or:
Redis
We first need to prove the failure timeline from production evidence.
The next lessons will move through:
evidence
↓
timeline
↓
root cause
↓
corrective design
in that order.
Incident Investigation Uses What We Already Built
This is where Module 11 becomes practical.
We now have:
structured logs
request IDs
metrics
health checks
These were not decorative production features.
They provide evidence.
We can investigate:
request R1
request R2
Order O1
Order O2
through logs and persisted state.
Start With the Two Orders
Suppose we identify:
Order O1
createdAt = T1
and:
Order O2
createdAt = T2
where:
T2
is shortly after:
T1
Both belong to:
Customer C1
Both have:
Product P1 × 2
Product P2 × 1
This is important evidence.
It still does not prove retry behavior.
Compare Historical Prices
Recall:
OrderItem.unitPriceCents
captures purchase-time Product price.
If both Orders have the same:
unitPriceCents
that may strengthen the suspicion that they originated close together.
But again:
same price
does not prove duplication.
We collect evidence without overstating it.
Inspect Inventory Effects
If both Orders committed, Inventory should reflect:
consumption for O1
+
consumption for O2
Example:
Initial:
P1 = 10
P2 = 5
Order O1 consumes:
P1 × 2
P2 × 1
State becomes:
P1 = 8
P2 = 4
Order O2 consumes the same quantities.
State becomes:
P1 = 6
P2 = 3
If that is what production state shows, the second Order was not just duplicated presentation.
Two independent transactional workflows executed.
This Means Transactions May Have Worked Correctly
That feels counterintuitive.
We have an incident.
Yet the database transaction may have behaved perfectly.
For O1:
Inventory consumed
Order persisted
OrderItems persisted
COMMIT
For O2:
Inventory consumed
Order persisted
OrderItems persisted
COMMIT
Each transaction preserves our local invariant.
The incident exists at a different boundary:
logical request identity
across retries
Correctness Has Multiple Layers
This is a major backend engineering lesson.
A system can be:
transactionally correct
but:
behaviorally incorrect
for a higher-level workflow.
Similarly, a database can be:
relationally valid
while representing:
wrong business outcome
Correctness is not one property.
Search Logs by OrderId
Structured logs may contain:
event=order_created
orderId=O1
customerId=C1
requestId=R1
and:
event=order_created
orderId=O2
customerId=C1
requestId=R2
This tells us something critical:
two different request executions
created two different Orders
Now we can investigate each request separately.
Search Request R1
For the first request we might find a sequence such as:
requestId=R1
request received
requestId=R1
event=order_created
orderId=O1
Then perhaps:
connection terminated
or:
application shutdown
or simply no evidence that the Customer received the response.
The exact production cause must come from actual evidence.
Do not invent missing log lines.
Search Request R2
The second request may show:
requestId=R2
request received
then:
requestId=R2
event=order_created
orderId=O2
then normal completion.
If R2 occurs shortly after R1 and matches the Customer report:
retry hypothesis becomes stronger
What If the First Request Has No Success Log?
Do not immediately conclude:
O1 was not committed
Remember the timing of logs and transactions.
A success log could:
not exist
fail to flush
occur before/after another failure
be missing due to instrumentation gaps
The database is authoritative for committed Order state.
Logs help reconstruct execution.
They do not replace persistence.
What If the Client Says "500"?
Even then, investigate.
Possible:
Order committed
response serialization failed
client received 500
Earlier in the course we explicitly recognized:
HTTP response formatting
as a different boundary from:
database transaction outcome
A 500 response does not automatically imply:
Order rolled back
What If the Client Says "Timeout"?
Even more ambiguous.
Timeout means:
client stopped waiting
It does not reveal exactly where the backend was.
Possible:
request never reached server
or:
request reached server
but transaction failed
or:
transaction committed
but response was delayed/lost
The incident investigation must determine which occurred.
Do Not Blame the Network Yet
The Customer's description may suggest:
network issue
But the system still has a responsibility to handle ambiguous retry situations safely if retry semantics are required.
Even if a network interruption triggered the incident:
network interruption
may be a trigger.
It is not necessarily the root cause of:
duplicate durable Orders
Root cause analysis must distinguish these.
Trigger vs Root Cause
Example:
Trigger:
HTTP response was not received.
Possible underlying system weakness:
Create Order had no way
to recognize retry of
the same logical operation.
These are not the same statement.
A strong incident review separates:
trigger
contributing factors
root cause
impact
Do Not Blame the Customer
Another weak conclusion:
Customer clicked twice.
Even if true, ask:
Was the operation expected
to tolerate retry?
Did our UX encourage retry?
Could the first result
have been unknowable?
Could an automated client
do the same thing?
Reliable backend design should not rely only on:
humans never retry
Do Not Blame the Frontend
A frontend may eventually help by:
disabling repeated submission
showing pending state
persisting an operation key
But the backend owns:
durable business side effects
A robust backend API must define the semantics of repeated side-effecting requests.
UI protection can supplement that.
It cannot be the only correctness mechanism.
Do Not Add an Arbitrary Retry Framework
Another reaction might be:
Maybe the server should automatically retry Create Order.
That can make the problem worse.
Create Order:
creates a new OrderId
consumes Inventory
Blindly rerunning the workflow can produce exactly the duplication we are investigating.
Retry safety requires:
operation identity
clear semantics
not simply:
try again
Do Not Delete One Order During Investigation
Operations might be tempted to:
DELETE duplicate Order
immediately.
But our database intentionally protects historical state.
We do not have a general Order deletion capability.
Deleting production records can also destroy evidence.
Incident response should first establish:
which Order is valid
what downstream effects occurred
whether payment happened
whether Inventory needs correction
before deciding remediation.
Do Not Manually "Fix Inventory" First
If both Orders consumed Inventory, changing:
available_quantity
manually before understanding Order state can create new inconsistency.
For example:
restore Inventory for O2
while O2 later becomes a legitimate paid Order.
Operational remediation must follow business truth.
Investigation comes first.
Scope the Impact
One Customer report does not tell us whether this is:
one incident
or:
systematic production behavior
We need to ask:
How many duplicate-looking Orders exist?
Did this start recently?
Is it associated with timeouts?
Is it associated with one client type?
Is it associated with one application version?
Are any duplicate Orders PAID?
How much Inventory was consumed?
Did any Product become unavailable incorrectly?
This is incident impact analysis.
Do Not Detect Impact Using a Naive Equality Rule
A query such as:
same customer
+
same items
+
close timestamp
can find:
candidates
It cannot automatically label them:
duplicates
because legitimate repeated purchases can look identical.
Candidate analysis may require:
logs
request timeline
support evidence
request metadata
Metrics May Reveal Scale
Our custom metrics may tell us:
order creation rate
technical failure rate
Built-in HTTP metrics may show:
POST /api/v1/orders latency
5xx rate
These can help answer:
Did something unusual happen
around the incident time?
Metrics detect patterns.
They generally do not identify:
which exact Order
because we deliberately avoided high-cardinality Order IDs in metric tags.
Logs Identify Individual Executions
This is exactly why our observability design separated:
metrics
from:
logs
Metrics:
How many?
How often?
How long?
Logs:
What happened to R1?
What happened to O1?
Incident response combines them.
Health May Rule Out Certain Hypotheses
Health history may tell us whether:
PostgreSQL readiness
was unhealthy around the time of the incident.
If PostgreSQL remained healthy:
database outage becomes less likely
as the trigger.
If readiness dropped:
database instability deserves investigation
Health does not prove the duplicate mechanism.
It contributes context.
CI Cannot Prevent Every Production Incident
Our CI may be completely green.
Every existing test may pass.
Why?
Because we had not yet specified:
retry same logical Create Order
after uncertain response
→ return/reuse same outcome
A test suite protects:
known requirements
known risks
known regressions
Production sometimes discovers:
new requirement
or:
missing scenario
This does not make testing useless.
It tells us how the test suite should evolve.
The Incident Must Become a Regression Test
Once root cause and desired semantics are established:
incident scenario
should become:
automated regression test
Then future refactoring cannot silently reintroduce the same failure.
That happens later in this module.
For now, preserve the scenario precisely.
Initial Incident Timeline
At the beginning of investigation, we can record a provisional timeline.
For example:
T1
Customer submits Create Order request R1.
T2
Backend processes R1.
T3
Order O1 appears in committed database state.
T4
Customer does not receive a successful response.
T5
Customer retries as request R2.
T6
Backend processes R2.
T7
Order O2 commits.
T8
Customer receives successful response for R2.
T9
Customer notices O1 and O2.
This is a working timeline.
Every point should later be tied to evidence.
Label Unknown Parts
For example:
T3
O1 committed.
Evidence:
database state / createdAt / logs.
T4
Customer did not receive success.
Evidence:
Customer report.
But:
Why did R1 response fail?
may remain:
UNKNOWN
Do not fill gaps with confidence you do not have.
Incident Documents Should Distinguish Evidence
A useful structure:
Observed
Confirmed
Suspected
Unknown
For example:
Confirmed
Two Orders exist.
Both belong to the same Customer.
Both consumed Inventory.
Reported
Customer did not receive
the first successful response.
Suspected
Second request was a retry
of the same logical operation.
Unknown
Exact reason first response
did not reach the Customer.
This keeps the investigation honest.
Severity
We do not have an organizational severity framework defined in this course.
So do not invent:
SEV-1
P0
Critical-7
without a real company convention.
We can still describe impact:
Customer can receive duplicate Orders.
Inventory can be consumed twice.
If duplicate Orders are later paid,
financial impact may occur.
That is enough to establish seriousness.
Potential Business Impact
Duplicate Orders can affect more than one table.
Possible consequences:
Inventory reduced twice
Customer confusion
Customer may pay twice
support workload
incorrect Order history
downstream operational confusion
Do not claim all of these occurred.
They are:
potential impact
that investigation should assess.
Payment Makes the Incident More Serious
If both Orders remain:
UNPAID
we still have a correctness issue.
If both become:
PAID
potential Customer financial impact exists.
Therefore investigation should check:
status of O1
status of O2
PaymentAttempt state
for each Order
Do not assume the duplicate Order incident is isolated from Payment.
Cancellation Is Not Automatically the Remedy
Could we simply cancel one duplicate Order?
Maybe.
But determine:
which one
whether it is still UNPAID
whether a Payment attempt is unresolved
whether cancellation is permitted
whether Inventory restoration is safe
Recall our established rule:
unresolved Payment attempt
can block cancellation
Incident remediation must respect existing business invariants.
Production Remediation and Permanent Fix Are Different
We may eventually need two tracks.
Immediate remediation
For affected Customers:
inspect Orders
determine valid business state
correct safely
Permanent corrective work
prevent equivalent retry scenario
from creating duplicate Orders
Do not confuse:
repair existing data
with:
prevent future incidents
Do We Need an Emergency Code Change?
Not necessarily.
First determine:
frequency
impact
reproducibility
safe mitigation
If incidents are frequent and severe:
urgent mitigation
may be appropriate.
If evidence is incomplete:
investigation
may be safer than rapidly shipping the wrong fix.
Incident response is risk management.
Potential Temporary Mitigations
Depending on evidence, a real team might consider:
support guidance
client retry guidance
temporary feature restriction
increased monitoring
manual review
But our source does not define a specific operational environment or mitigation.
We will not invent one.
The course focuses on:
understanding
designing
implementing
testing
the corrective backend behavior.
Create the Incident Investigation Questions
Before the next lesson, our investigation checklist is:
1. Confirm both Orders exist.
2. Confirm both transactions committed.
3. Compare Customer and OrderItems.
4. Find request IDs related to both Orders.
5. Reconstruct request sequence.
6. Determine whether R2 was a retry.
7. Determine what happened to R1 response.
8. Check Inventory effects.
9. Check Payment state.
10. Assess whether other Customers
experienced similar behavior.
11. Determine whether deployment,
database health, or network behavior
contributed.
12. Identify the system condition
that allowed one logical operation
to create two Orders.
Only then should we design the fix.
What We Should Not Change Yet
At this stage, do not add:
Idempotency-Key
yet.
Do not create:
idempotency_requests
table yet.
Do not add:
Redis
yet.
Do not add:
unique index
yet.
Do not change:
OrderId generation
yet.
Do not implement:
automatic retries
yet.
Do not change:
POST /api/v1/orders
contract yet.
Those decisions belong after root cause and requirements are explicit.
The Key Engineering Distinction
This incident teaches a distinction that is easy to miss:
database atomicity
is not:
request idempotency
Transaction gives us:
one execution
→ all local database changes
commit together
Idempotency would give us something like:
same logical operation
executed repeatedly
→ one intended business effect
These are different guarantees.
Another Important Distinction
Likewise:
HTTP response
is not:
business transaction
The server can:
commit successfully
while:
client observes failure
Reliable backend design must reason across both boundaries.
Why This Incident Is Useful
Until now we designed failure handling mostly from known requirements.
This incident forces us to experience another part of backend engineering:
Production behavior can expose missing assumptions that were invisible during the first implementation.
A strong engineer must be able to:
understand existing system
collect evidence
avoid premature conclusions
identify the failed assumption
design a compatible change
protect existing data and clients
add regression coverage
That skill is more important than merely knowing a new annotation.
Backlog Evolution
Our original backlog ended at:
BACKEND-128
But production systems do not have a truly finished backlog.
Now reality adds:
BUG-201
Prevent duplicate order creation
under identified retry scenario
This is expected.
A backlog represents:
current understanding
not eternal truth.
Do Not Rewrite History
An incident review should not say:
We should obviously have known this.
The earlier design explicitly recognized the possibility but intentionally did not invent an idempotency contract without a requirement.
A useful review asks:
What did we know?
What assumption did we make?
What did production teach us?
What should change now?
This produces learning rather than blame.
Blameless Does Not Mean Responsibility-Free
Avoid:
nobody did anything wrong
so nothing needs to change
We still need to identify:
system weakness
missing requirement
missing test
missing operational signal
process improvement
Blameless investigation means:
focus on system conditions
rather than personal blame
Corrective action remains essential.
Initial Incident Record
At the end of this lesson, our incident record can begin like this:
Incident:
Duplicate Order Created After Retry
Status:
Under Investigation
Impact:
At least one Customer observed two persisted Orders
after retrying an Order creation request whose first
response appeared unsuccessful.
Known:
- Both Orders exist in PostgreSQL.
- Both Orders have distinct UUIDs.
- Both belong to the same Customer.
- Both contain equivalent requested items.
- Both committed Inventory effects.
- Customer reports retrying after an uncertain first response.
Unknown:
- Exact reason the first response was not received.
- Whether similar incidents affected other Customers.
- Whether both Orders entered Payment processing.
- Exact system timeline across both requests.
Suspected Risk:
Create Order currently has no request-level identity
that allows the backend to recognize a retry of the
same logical operation.
Next Action:
Reconstruct the incident using persisted state,
structured logs, request IDs, metrics, and runtime evidence.
Notice:
Suspected Risk
is not yet labeled:
Root Cause
That comes after investigation.
Engineering Principle
The first principle:
During an incident, establish what happened before deciding how to fix it. Production symptoms, hypotheses, and confirmed facts are not the same thing.
The second:
A successful database transaction and a failed client experience can occur simultaneously. Transaction atomicity protects database consistency; it does not tell the client whether its response was delivered.
The third:
Repeated side-effecting requests require explicit semantics if the system must treat them as one logical operation. Similar payloads, timestamps, request IDs, and UI protections are not substitutes for a deliberate operation identity.
And:
Production incidents are inputs to system design. The backlog, tests, documentation, and architecture should evolve as the system teaches us something new.
Summary
In this lesson, we established that:
- Module 13 begins with a production incident rather than a new greenfield feature.
- A Customer reports two Orders after retrying an uncertain Create Order request.
- Two distinct Orders exist in PostgreSQL.
- Both Orders may be individually valid.
- Both may have consumed Inventory correctly.
- An incident does not automatically imply transaction corruption.
- Production investigation starts with facts rather than proposed fixes.
- Persistent committed state is the primary source of truth for database outcomes.
- Client-visible failure does not prove the server transaction failed.
- A transaction may commit before the HTTP response is lost.
- PostgreSQL atomicity protects one Create Order execution.
- PostgreSQL does not know whether two requests represent one logical Customer intention.
- The current Create Order contract generates a new OrderId for each execution.
- Equivalent Order payloads do not reliably identify a retry.
- Legitimate Customers may intentionally place identical Orders.
- Time-window deduplication can incorrectly suppress legitimate purchases.
- UI button disabling does not provide backend retry safety.
X-Request-Idis operational correlation, not idempotency identity.OrderIddoes not solve the retry problem because the server generates it and a client may never receive it.- Payment idempotency semantics do not automatically make Create Order idempotent.
POSTcreation is not automatically retry-safe.- The original design intentionally did not invent request-level Create Order idempotency before a concrete requirement existed.
- Production has now provided such a concrete scenario.
- Structured logs, request IDs, metrics, health history, and PostgreSQL state provide investigation evidence.
- Candidate duplicate Orders must not automatically be classified using only Customer/items/time similarity.
- Metrics identify patterns while logs identify individual executions.
- CI can be green while a previously unspecified production scenario remains unprotected.
- Once the incident is understood, it must become regression coverage.
- An incident timeline should distinguish confirmed facts, reports, hypotheses, and unknowns.
- We should not invent an organizational severity scheme that the course has not defined.
- Potential impact includes Inventory, Customer experience, and possibly Payment, but actual impact must be verified.
- Existing Payment and cancellation invariants still apply during remediation.
- Repairing existing production state and preventing future recurrence are separate tasks.
- We should not add an idempotency mechanism before completing the investigation.
BUG-201now represents the corrective backlog work created by production reality.
Next lesson:
Investigating With Production Signals
We will take the incident from:
Customer says:
"I got two Orders."
to an evidence-backed timeline using:
PostgreSQL state
Order createdAt values
structured logs
requestId
orderId
customerId
PaymentAttempt state
HTTP/application metrics
health history
and learn how to distinguish:
evidence
correlation
inference
root cause
without jumping directly to a fix.