Backend Development with Java & Spring BootLesson 101 / 120

Handling Production Incident

Investigating With Production Signals

ReadingPreview

You are viewing a free preview lesson.

In the previous lesson, production gave us an incident:

Customer intended to create one Order.

First request appeared unsuccessful.

Customer retried.

Two Orders were eventually visible.

We deliberately did not jump straight to a fix.

Now we investigate.

Our goal is:

Reconstruct what happened using independent production signals before declaring a root cause.

This is where the observability work from Module 11 becomes operationally useful.

We have several evidence sources:

PostgreSQL state

structured logs

request IDs

Order IDs

CustomerId

PaymentAttempt state

HTTP metrics

application metrics

health signals

deployment/runtime information

Each tells us something different.

No single signal should automatically become the entire incident story.

Evidence Before Explanation

A weak incident investigation starts with:

We think the retry created the duplicate.

Let's implement idempotency.

A stronger investigation starts with:

What can we prove?

What can we correlate?

What are we inferring?

What remains unknown?

That distinction prevents us from building a fix for the wrong failure mode.

Our Investigation Questions

We need to answer:

Did both Orders actually commit?

Which HTTP requests created them?

Were those requests from the same Customer?

Were their requested items equivalent?

How far apart were they?

Did the first request complete normally?

Did the first transaction roll back?

Was there a database incident?

Was there an application restart?

Was there a Payment involved?

Was the second request truly a retry?

Are there more affected Customers?

The evidence should narrow these questions progressively.

Signal 1 — PostgreSQL

Start with the durable system of record.

Why?

Because our earlier transaction design established:

Committed database state is authoritative for whether an Order actually exists.

The Customer's browser may say:

request failed

But PostgreSQL tells us whether:

Order committed

Inspect the Candidate Orders

Suppose support identifies two candidate Orders:

O1 =
36a25516-2af2-4644-b55d-d0cabf389c9e

and:

O2 =
b29c51cc-f75a-46ae-a74c-b4c342547a20

We inspect:

orders

order_items

For O1:

customerId = customer-742

status = UNPAID

createdAt = T1

For O2:

customerId = customer-742

status = UNPAID

createdAt = T2

Then compare OrderItems.

Suppose both contain:

Product P1 × 2
unitPriceCents = 4990

Product P2 × 1
unitPriceCents = 1990

This gives us strong evidence that the Orders are related.

It still does not prove they are duplicates.

Why Database Similarity Is Not Enough

Two Orders can legitimately share:

same Customer

same Products

same quantities

same prices

A Customer may intentionally place the same purchase twice.

Therefore:

same content

is:

correlation evidence

not:

operation identity

We need more.

Compare createdAt

Suppose:

O1.createdAt = T1

and:

O2.createdAt = T1 + 12 seconds

Now the retry hypothesis becomes stronger.

But even:

12 seconds apart

does not prove retry semantics.

Time proximity is useful evidence.

It is not a safe deduplication rule.

Verify Both Orders Actually Committed

Because both Orders are queryable after the incident:

O1 exists

O2 exists

we know both transactions reached durable committed state.

This immediately rules out one hypothesis:

The first Order rolled back
and the UI accidentally showed it.

If O1 is really in PostgreSQL after the incident:

O1 committed

That matters enormously.

Inspect Inventory State

Our business model says:

successful Order creation
→ Inventory decreases

So inspect Inventory consequences.

Suppose before both Orders, the relevant Inventory was approximately:

P1 = 10
P2 = 5

After O1:

P1 = 8
P2 = 4

After O2:

P1 = 6
P2 = 3

If production evidence confirms both mutations, then:

two Create Order workflows committed

This is not merely duplicated presentation.

Do Not Reconstruct Inventory From Guesswork

Be careful.

Our current schema stores current Inventory:

available_quantity

not a complete Inventory movement ledger.

Therefore we cannot always reconstruct historical quantities perfectly from current state alone.

Use:

Order state

logs

known events

current Inventory

other operational evidence

carefully.

Do not pretend the schema contains an Inventory audit history when it does not.

Signal 2 — Structured Logs

Now search logs using:

orderId

For O1 we may find:

event=order_created
orderId=O1
customerId=customer-742
requestId=R1

For O2:

event=order_created
orderId=O2
customerId=customer-742
requestId=R2

This is powerful.

We now know:

O1 was associated with request R1

O2 was associated with request R2

So the two Orders did not arise from one accidental duplicate persistence operation inside a single request.

They came from:

two request executions

Request IDs Clarify Execution Boundaries

This is exactly why we introduced:

X-Request-Id

Earlier.

A request ID answers:

Which logs belong to
one HTTP request execution?

It does not answer:

Which requests represent
the same logical Customer intention?

But during an incident, knowing that:

R1 ≠ R2

is already valuable.

Search All Logs for R1

Now query centralized logs for:

requestId=R1

We may find evidence such as:

request accepted

then:

event=order_created
orderId=O1

and perhaps later:

response handling failure

or:

connection closed

or:

application shutdown

depending on what actually happened.

The exact evidence depends on production.

Do not invent a line merely because it would make the story convenient.

Search All Logs for R2

Then search:

requestId=R2

Potentially:

request accepted

then:

event=order_created
orderId=O2

followed by a normal completion.

Now we can compare:

R1 timeline

R2 timeline

rather than looking at all logs globally.

Structured Fields Beat Message Searching

Imagine we had only:

"Order created successfully"

repeated thousands of times.

Investigation would be difficult.

Because we structured logs with fields such as:

requestId

orderId

customerId

event

we can move between dimensions:

Order
→ request

request
→ all logs

Customer
→ candidate activity

This is why logging design is part of production engineering.

But Logs Are Not the Source of Truth

Suppose we do not find:

event=order_created
orderId=O1

Does that prove O1 did not commit?

No.

If PostgreSQL contains O1:

O1 exists

The missing log may mean:

instrumentation gap

log delivery problem

process interruption

log timing difference

Never elevate logs above authoritative business persistence.

Logs explain execution.

They do not redefine committed state.

Success Log Timing Matters

Recall from structured logging:

log says "Order created"

can be misleading if emitted before transaction commit.

When investigating production:

database committed state

is stronger evidence of durable business success than:

one success log

This is why observability semantics need careful interpretation.

Signal 3 — HTTP Metrics

Metrics answer aggregate questions.

For the incident time window, inspect:

POST /api/v1/orders

behavior.

Questions:

Did latency increase?

Did 5xx responses increase?

Did request volume spike?

Was there a general application problem?

Was this one isolated request?

Suppose the normal request latency was stable, but around T1:

one small increase in 5xx

appears.

That can support an investigation.

It cannot tell us:

which exact Customer request failed

because we intentionally do not place:

customerId

orderId

requestId

into metric labels.

This Is Good Metric Design

During an incident it may feel tempting to ask:

Why didn't we put orderId
into metrics?

Because that would create high-cardinality time series.

Metrics answer:

Is there a pattern?

Logs answer:

What happened to O1?

Our telemetry remains complementary.

Check Order Creation Metrics

If we added:

order.creation

metrics, inspect outcomes around the incident.

Perhaps:

success

increments twice.

That would support:

the application considered
both Order workflows successful

But again:

metric count

does not identify which specific Orders were involved.

Use metrics for pattern confirmation.

What Would a 5xx Mean?

Suppose HTTP metrics show:

POST /api/v1/orders
→ 500

around R1.

Do not immediately conclude:

Order creation failed.

We already know O1 committed.

A 500 could theoretically occur:

after transaction commit

during a later HTTP-layer failure.

For example:

serialization failure

response handling failure

other post-commit exception

The exact cause requires logs/evidence.

This is a critical production lesson:

HTTP response status

and:

transaction outcome

can diverge.

Signal 4 — Application Health

Look at:

liveness

readiness

around T1.

Questions:

Was the application ready?

Was PostgreSQL healthy?

Did readiness drop?

Did the process restart?

Suppose readiness remained:

UP

throughout the incident.

That makes a broad PostgreSQL outage less likely.

It does not rule out:

one transient connection issue

or:

HTTP response problem

But it helps narrow the investigation.

If Readiness Dropped

Suppose instead:

readiness
→ DOWN

because PostgreSQL became unavailable.

Then database instability becomes an important contributing factor.

However:

database outage

still does not automatically explain:

two committed Orders

We must still connect events.

A dependency incident could trigger retries.

The missing idempotency behavior may still be what converts the retry into duplication.

Signal 5 — Runtime / Deployment Events

Check whether something happened to the application instance around R1:

container restart

deployment rollout

process termination

instance replacement

startup event

Why?

One possible timeline is:

Order commits
    ↓
application process terminates
    ↓
response never reaches client

If runtime records show a restart immediately after O1 commit:

strong contributing evidence

But remember:

The restart may explain why the response was lost; it does not by itself explain why the retry created another Order.

Trigger vs System Weakness

Suppose we confirm:

deployment terminated instance
after database commit
before response delivery

Then:

deployment timing

may be the trigger.

But if the system cannot recognize retry of the same logical operation:

missing retry identity

is the system condition that allowed another Order to be created.

Incident analysis needs both.

Signal 6 — Payment State

Inspect both Orders.

Are they:

UNPAID?

Did either have:

PaymentAttempt?

Questions:

Was O1 paid?

Was O2 paid?

Was either Payment outcome unknown?

Did the Customer attempt payment
for only one Order?

Are both still cancellable?

This determines incident impact and possible remediation.

Why Payment State Matters Even Though Incident Is Create Order

Create Order and Payment are separate workflows.

But production impact can cross workflows.

Example:

duplicate Order O1

duplicate Order O2

If only O2 is paid:

O1 may be a removable/cancellable duplicate candidate

subject to existing rules.

If both are paid:

financial impact may have occurred

and the incident becomes significantly more serious.

Never Guess Which Order Is the "Duplicate"

It is tempting to say:

second Order = duplicate

But imagine:

O1 committed
Customer never saw it

O2 succeeded visibly
Customer then paid O2

Operationally, O1 might be the unwanted business record.

Chronologically:

O2 is later

but business remediation may treat:

O1

as the unintended leftover.

"Duplicate" is a business interpretation, not simply:

later row

Signal 7 — Security / Customer Identity

Confirm both Orders belong to the same authenticated:

CustomerId

Our Create Order API does not accept:

customerId

from request body.

Ownership comes from authenticated identity.

Therefore if both Orders contain the same CustomerId:

they were created under the same authenticated Customer context

assuming no broader security compromise.

This helps rule out:

another Customer accidentally created one

Do Not Search for a Local Customer Entity

Remember:

there is no Customer table

in our application.

Customer identity comes from the external authentication capability and is stored as:

orders.customer_id

Incident investigation must respect actual architecture.

Do not invent:

CustomerRepository

for analysis.

Reconstructing a Working Timeline

After combining evidence, we might produce a provisional timeline.

The following is a course scenario example, not source-provided production data:

10:14:03.120
Request R1 received:
POST /api/v1/orders
Customer C1

10:14:03.142
CreateOrderUseCase begins.

10:14:03.168
Products validated.

10:14:03.190
Inventory consumed.

10:14:03.212
Order O1 persisted.

10:14:03.219
Database transaction commits.

10:14:03.xxx
Client does not receive confirmed success.

10:14:15.441
Request R2 received:
POST /api/v1/orders
Customer C1

10:14:15.514
Order O2 commits.

10:14:15.530
Client receives 201 Created for O2.

This becomes useful only when each step is backed by actual evidence.

Annotate the Evidence Source

A strong timeline should say where each fact came from.

Example:

TimeEventEvidence
T1R1 acceptedaccess/application log
T2O1 createdPostgreSQL + structured log
T3O1 committeddurable DB state
T4response uncertainCustomer report / runtime evidence
T5R2 acceptedaccess/application log
T6O2 createdPostgreSQL + structured log
T7R2 returned successHTTP/client evidence

This helps reviewers distinguish:

fact

from:

inference

"Commit" Can Be Hard to Observe Directly

Application logs may not literally contain:

COMMIT succeeded

for every transaction.

But if:

O1 persists in PostgreSQL after incident

we know its transaction committed.

That is enough.

We do not need SQL-level transaction logging for every request.

Avoid Turning on Maximum Logging During the Incident Without Thought

Someone may suggest:

enable DEBUG everywhere

or:

enable Hibernate SQL logging

in production.

This can create:

huge volume

performance overhead

sensitive-data exposure

harder analysis

Use targeted diagnostics if needed.

Our structured production signals should already answer many questions.

Search by Customer Carefully

Searching:

customerId=C1

may reveal all recent Orders from that Customer.

This helps identify:

O1

O2

perhaps other candidate repeats

But Customer activity can be legitimate.

Do not automatically label every similar event as incident-related.

Search by Event

Useful structured queries:

event=order_created
customerId=C1

or:

orderId=O1

or:

requestId=R1

Each query answers a different question.

This is better than searching arbitrary phrases such as:

"created order"

through unstructured text.

Determine Whether R2 Was Actually a Retry

What evidence supports that?

Possibilities:

same Customer

same item composition

close timing

support report

same client session

client logs if available

frontend telemetry if available

Our backend alone may not know:

logical client intent

because that is exactly the missing operation identity.

We may reach a high-confidence conclusion without mathematically proving intent.

Incident reviews should state that honestly.

Correlation Is Not Causation

Suppose:

R2 occurs 12 seconds after R1

and creates an identical Order.

That strongly correlates.

But correlation alone does not mean:

R2 must have been automatic retry

It could have been:

manual second click

frontend retry

mobile retry

user refresh/resubmit

For the backend fix, the exact retry mechanism may not matter.

The relevant behavioral requirement may simply be:

same logical operation
must be repeatable safely

Check for Other Candidate Incidents

One Customer establishes the bug possibility.

Impact analysis asks whether it happened elsewhere.

We might search for candidate patterns such as:

same customer

equivalent item set

close createdAt

But the result is:

candidate list

not:

confirmed duplicate list

Each candidate may require additional evidence.

Do Not Run an Unsafe Giant Production Query

Incident urgency does not justify:

full-table cross join

against production.

Use:

indexes

bounded time windows

existing query capabilities

safe read queries

and coordinate with database operations practices.

Investigation itself must not become an outage.

Our Existing Order Index Helps

We already have:

CREATE INDEX idx_orders_customer_created_id
ON orders (
    customer_id,
    created_at DESC,
    id DESC
);

This supports:

recent Orders for one Customer

efficiently.

That is useful for focused incident investigation.

We do not need to invent a new index immediately.

Be Careful Querying OrderItems

To compare candidate Orders, we may inspect:

product_id

quantity

unit_price_cents

But remember:

OrderItem

has composite identity:

(order_id, product_id)

There is no independent:

OrderItemId

Investigation should follow the actual schema.

Check Product State Only if Relevant

Current Product state might differ now from the state when O1 and O2 were created.

For example:

Product now inactive

does not tell us it was inactive during Order creation.

Order creation already captured historical:

unitPriceCents

and existing Orders remain valid even if Product state changes later.

Do not use current Product state to rewrite incident history.

Historical Order Data Is Valuable Evidence

Because OrderItem preserves:

purchase-time unitPriceCents

we can understand what each Order represented at creation time.

This is one benefit of our historical modeling decision.

If we had only referenced current Product price:

incident reconstruction could become inaccurate

after later Product changes.

Check Deployment Version

If operational metadata tells us which application image handled R1 and R2, record it.

Possible:

R1
→ application version A

R2
→ application version B

during a rollout.

That may matter.

Or:

both handled by version A

which rules out certain deployment-change hypotheses.

Artifact traceability from CI/containerization now becomes useful.

Deployment Difference Does Not Automatically Mean Deployment Bug

Suppose R1 occurred during a rollout.

Do not conclude:

deployment caused duplicate Orders

without evidence.

The rollout may explain:

lost response

while the missing retry safety explains:

second durable business effect

Multiple contributing conditions can coexist.

Incident Root Cause Can Be Multi-Layered

A useful model:

Trigger
→ why the incident began

System weakness
→ why trigger created harmful outcome

Detection gap
→ why we did not know earlier

Impact
→ what users/business experienced

For example, later we may conclude:

Trigger:
client did not receive R1 response

System weakness:
Create Order had no logical retry identity

Detection gap:
no metric specifically surfaced repeated logical creations

Impact:
two committed Orders and double Inventory consumption

But we should not finalize this yet.

What If Logs Show R1 Returned 201?

Suppose server access logs show:

R1
→ 201

but Customer says no success was received.

That is possible.

Server-side:

response generated / written

does not always prove:

client consumed response

Network intermediaries may still fail.

The backend cannot derive client certainty from its own status code alone.

What If R1 Shows 500?

Suppose server logs show:

R1
→ 500

while O1 exists.

Now we know something important:

failure happened after
the business transaction became durable

or the error mapping/response layer diverged from business outcome.

This is a valuable root-cause branch to investigate.

What If R1 Has No Completion Record?

Possible:

process terminated

connection interrupted

logging gap

Then use other signals:

runtime restart

container logs

deployment events

database state

Incident investigation works by triangulation.

Triangulation

Triangulation means combining independent signals.

Example:

PostgreSQL:
O1 exists.

Application log:
R1 created O1.

Runtime:
instance terminated seconds later.

Customer:
did not receive success.

R2 logs:
same Customer created equivalent O2.

Together, these tell a much stronger story than any single source.

Never Let One Dashboard Become "Truth"

Dashboards are views over telemetry.

They can have:

sampling

aggregation

query mistakes

delayed data

missing series

Always understand what the underlying signal means.

Incident response requires engineering interpretation.

Health Signals Have Limited Scope

Remember our health design:

readiness
→ can serve normal traffic

liveness
→ should process remain running

Health cannot tell us:

was R1 idempotent?

A healthy application can still have a business correctness bug.

This incident may happen while:

health = UP

the entire time.

That does not contradict the incident.

Metrics Have Limited Scope Too

Metrics can show:

two successful Order creations

But metrics may not know they represent:

one intended purchase

because that logical operation identity does not exist yet.

Observability cannot infer a concept the application never modeled.

This is an important design insight.

Missing Data Is Itself Information

Suppose we realize:

we cannot reliably identify
whether R1 and R2 represented
the same logical operation

from backend data alone.

That is not merely an observability failure.

It may reveal the core modeling gap:

logical operation identity
was never represented

This directly informs the corrective design.

Distinguish Observability Gap From Business Model Gap

An observability gap would be:

requestId existed
but was not emitted in logs

A business model/API gap may be:

no idempotency/operation key existed
for Create Order

Adding more logging does not solve the second problem.

We must classify the gap correctly.

Do Not "Fix" the Incident With More Logs Only

We may improve observability after the incident.

But:

better logs

do not stop duplicate Orders.

They help us detect and investigate them.

Corrective work must address:

business operation semantics

if that is what failed.

Incident Evidence Worksheet

A practical investigation record might look like:

Customer:
customer-742

Candidate Orders:
O1
O2

O1:
status = UNPAID
createdAt = T1
requestId = R1

O2:
status = UNPAID
createdAt = T2
requestId = R2

Item comparison:
equivalent

Order IDs:
different

Request IDs:
different

Inventory:
consistent with two successful consumptions

Payment:
no confirmed Payment yet

Database health:
no broad outage observed

Runtime:
under investigation

Client report:
R1 appeared unsuccessful
R2 was retry

Other affected Customers:
under investigation

This gives everyone a common factual baseline.

Update the Incident Document

Our incident document can now evolve.

Status

Investigating

Confirmed

Two distinct Orders committed.

Both belong to the same Customer.

Both contain equivalent items.

Each Order is associated with
a different HTTP request.

Both produced Inventory effects.

Reported

Customer did not observe success
for the first request.

Customer retried.

Under Investigation

Why first response was uncertain.

Whether the application restarted.

Whether a network intermediary timed out.

Whether other Customers were affected.

Emerging System Gap

Create Order has no stable
logical-operation identifier
shared across retries.

Still do not label that:

final root cause

until the timeline is complete.

Impact Assessment

Now ask:

How many Customers?

How many candidate duplicate Orders?

How many became PAID?

How much Inventory was affected?

When did this first occur?

Is the issue ongoing?

These answers determine incident urgency.

Do not define severity using an invented framework.

Use measurable impact.

Detection

How was the incident discovered?

In our scenario:

Customer/support report

That is important.

It suggests the system may not have automatically detected:

logical duplicate creation

because no operation identity existed to measure.

This may become a corrective observability action later.

But Do Not Invent a "Duplicate Order Metric" Yet

How would the application know two Orders are duplicates without defining:

same logical operation

first?

A metric based on:

same Customer + same items + 30 seconds

would produce false positives.

Correct modeling must come before reliable telemetry.

Investigation Should Be Reproducible

A strong engineering investigation should be something another engineer can follow.

Record:

queries used

log filters

relevant request IDs

Order IDs

time window

application version

important observations

Do not rely solely on:

I looked at the dashboard
and it seemed weird.

Incident documents preserve engineering reasoning.

Preserve Relevant Evidence

If logs have limited retention, record references to:

R1

R2

O1

O2

time window

application version

in the incident document.

Do not paste:

secrets

tokens

personal data

huge log dumps

unnecessarily.

Preserve useful evidence safely.

Do Not Copy Sensitive Data Into Incident Docs

Our observability safety rules still apply.

Avoid storing:

Authorization headers

access tokens

Payment Provider credentials

raw personal information

just because the document is internal.

Use safe identifiers.

When Do We Have Enough Evidence?

We do not need perfect knowledge of every packet that crossed the network.

We need enough evidence to explain:

what durable business effects occurred

which request executions caused them

why the second execution was allowed

what system assumption was missing

The exact reason R1 response disappeared may remain partially unknown.

We can still fix the repeat-execution weakness.

Root Cause Does Not Require Perfect Trigger Knowledge

Suppose we cannot determine whether R1 failed because of:

client disconnect

proxy timeout

instance termination

But we can prove:

R1 committed O1

Customer did not know outcome

R2 executed same intended operation

backend had no retry identity

R2 committed O2

That may be enough to identify the relevant system weakness.

Do not block corrective engineering forever because one network detail is unknowable.

But Preserve the Unknown

The incident review should say:

Exact mechanism that prevented
the Customer from observing R1 success:
not conclusively determined.

That is stronger engineering than inventing certainty.

What We Have Narrowed Down

At this stage, assume our investigation confirms:

O1 committed.

O2 committed.

R1 created O1.

R2 created O2.

R1 and R2 were separate HTTP executions.

Both belonged to the same Customer.

Payload/business intent was equivalent.

Customer reports R2 was a retry.

No evidence shows transaction corruption.

No evidence shows duplicate persistence
inside a single transaction.

This dramatically narrows the problem.

What This Rules Out

We can likely rule out:

JPA inserted the same Order twice

database duplicated one INSERT

one transaction accidentally persisted
two Orders

OrderId collision

Inventory rollback failure

because:

two distinct request IDs

two distinct OrderIds

two independently valid committed workflows

fit the evidence better.

What Remains

The central question becomes:

Why did the application treat R2 as a completely new Order creation when the Customer considered it a retry of R1?

The answer is beginning to emerge:

the server had no durable identity
for the logical Create Order operation
across HTTP retries

The next lesson will formalize this as the incident timeline and root cause.

Engineering Principle

The first principle:

Use multiple production signals to reconstruct behavior. PostgreSQL tells you what committed, logs tell you what executed, request IDs group one execution, metrics reveal patterns, and health/runtime signals provide environmental context.

The second:

Do not ask a telemetry signal to answer a question it was not designed for. Metrics do not identify individual Orders, health does not prove business correctness, and logs do not override committed database state.

The third:

An incident timeline must separate evidence from inference. If a detail cannot be proved, mark it unknown rather than filling the gap with a convenient story.

And:

Sometimes investigation reveals that the missing information is not merely an observability problem—it is evidence that the business/API model lacks a concept the system needs, such as durable logical-operation identity across retries.

Summary

In this lesson, we established that:

  • Incident investigation begins with evidence rather than proposed fixes.
  • PostgreSQL is the authoritative source for committed Order state.
  • Two persisted Orders prove two durable business effects occurred.
  • Equivalent Customer, items, and timestamps strongly correlate Orders but do not automatically prove duplication.
  • Current Inventory state can support investigation but is not an Inventory movement ledger.
  • Structured logs let us map Order IDs to request IDs.
  • Different request IDs prove there were separate HTTP executions.
  • Request IDs identify executions, not logical Customer operations.
  • Logs help reconstruct behavior but do not override database truth.
  • Missing logs do not mean a committed Order never existed.
  • HTTP metrics reveal aggregate latency, status, and request patterns.
  • High-cardinality identifiers correctly remain out of metric labels.
  • A 500 response does not necessarily mean the Order transaction rolled back.
  • Health signals help assess environmental dependency state but cannot prove business correctness.
  • Runtime/deployment events can explain lost-response triggers.
  • A deployment restart may be a trigger without being the underlying reason duplicate business effects were possible.
  • Payment state must be checked because duplicate Orders can create financial impact.
  • The chronologically later Order is not automatically the business record that should be removed.
  • Customer identity investigation must follow the actual architecture—there is no local Customer aggregate.
  • Timelines should annotate their evidence sources.
  • Exact transaction commit may be inferred confidently from durable persisted state.
  • Investigation should not enable noisy or sensitive logging indiscriminately.
  • Candidate duplicate searches must remain candidates, not automatic classifications.
  • Existing database indexes can support focused investigation.
  • Current Product state should not rewrite historical Order interpretation.
  • Historical unitPriceCents helps preserve incident evidence.
  • Deployment artifact identity can help correlate incidents with application versions.
  • Trigger, system weakness, detection gap, and impact are different incident concepts.
  • A server-side 201 does not prove the client observed success.
  • A server-side 500 with a committed Order would indicate a post-commit/HTTP-boundary divergence.
  • Missing completion records require triangulation with other signals.
  • No single dashboard should be treated as absolute truth.
  • Observability cannot infer a logical operation identity the application never modeled.
  • Missing logical operation identity may represent an API/business-model gap rather than merely a logging gap.
  • Better logging alone would not prevent duplicate Orders.
  • Incident evidence should be documented reproducibly and safely.
  • We do not need perfect knowledge of the precise network trigger before fixing a proven retry-safety weakness.
  • Unknown details should remain explicitly unknown.
  • The evidence now points away from transaction corruption and toward repeated execution of a non-idempotent Create Order operation.

Next lesson:

Reconstructing the Failure Timeline and Finding the Root Cause

We will move from:

two committed Orders
+
two request executions
+
one reported logical Customer operation

to a precise failure model:

R1 begins
    ↓
O1 commits
    ↓
client outcome becomes uncertain
    ↓
R2 retries same logical operation
    ↓
server has no durable retry identity
    ↓
O2 commits

Then we will distinguish:

trigger

root cause

contributing factors

detection gap

impact

before designing the idempotency fix.