Handling Timeouts and External Failures

In the previous lesson, we built the chain PaymentService → ProviderPaymentService → ProviderPaymentClient → external payment provider, and established that a confirmed payment decline is not the same as a provider communication failure. External systems fail in ways that do not exist inside one PostgreSQL transaction: the provider declines the payment, returns 500, cannot be reached, times out, sends a malformed response or an unknown status, or processes the payment while the response is lost. These scenarios cannot all be treated as "payment failed". Some give us a known business outcome; others mean "we do not know what happened externally". The goal of this lesson is:

Classify external-service failures according to what the application actually knows, so confirmed payment outcomes, infrastructure failures and uncertain outcomes are never collapsed into one misleading result.

External Calls Change Failure Semantics

With local PostgreSQL work, a transaction gives us strong information: a committed UPDATE inventory is committed, a rolled-back one is not. An external provider is different. Between our backend and the provider's database sit several independent systems: our process, our network connection, the provider's edge, the provider's application and the provider's persistence. A failure at one layer does not necessarily tell us what happened at another.

The Most Important Question

When an external call fails, do not ask "which exception was thrown?" first. Business behaviour depends on a different question:

What do we actually know about the external side effect?

A useful classification has five categories, each with a different meaning:

1. Confirmed success
2. Confirmed business failure (a decline)
3. Technical failure before external execution is known
4. Uncertain outcome
5. Invalid or unexpected provider response

Confirmed Success and Confirmed Decline

The provider explicitly confirms that the payment succeeded, and the integration translates this into PaymentResult.succeeded(providerPaymentId). Only then may the application later call order.markPaid(). We never infer success merely from "no exception was thrown" or from HTTP 200, unless the provider contract says so: a provider could respond HTTP 200 with a body that says the payment was rejected. Transport success and payment success remain different concepts.

When the provider clearly declines the payment, the integration has communicated successfully and received a definitive business result: the payment did not succeed. The application represents this as PaymentResult.declined(reason), and the Order remains UNPAID. Provider-defined cases such as "payment not approved" or "payment method rejected" depend on the provider contract, but share one shape: the provider communicated normally and the outcome is definitively unsuccessful. That is an application result, not a system error. A decline also does not make the Order CANCELLED; cancellation is a separate customer action with inventory restoration consequences.

A Technical Failure Is Not a Decline

Now suppose the integration cannot communicate normally: DNS resolution fails, the connection is refused, TLS negotiation fails, the provider returns an unusable 5xx, or serialization fails before the request is sent. The system could not obtain a normal payment result, which is not the same as "the provider rejected the payment". The tempting shortcut is wrong:

try {
    return providerClient.pay(request);
} catch (Exception exception) {
    return PaymentResult.declined("provider error");
}

This translates "provider unavailable" into "payment declined". That is false, and it hides a production outage, a network issue, a configuration error or an integration bug behind a business result. A provider decline becomes a PaymentResult; a technical failure stays an exception:

Expected business outcomes and infrastructure failures must remain distinct.

Uncertain Outcome: A Timeout Means Lack of Knowledge

The most dangerous category is the uncertain outcome. Consider this timeline:

1. Backend sends payment request.
2. Provider receives request.
3. Provider successfully charges the customer.
4. Provider sends response.
5. Network connection breaks.
6. Backend times out.

Our backend knows only that it did not receive a confirmed response. It does not know that the payment failed; it may already have succeeded. This is the central principle of the lesson:

A timeout tells us that our waiting period expired. It does not necessarily tell us what happened at the provider.

A timeout can occur before the provider receives the request, while it processes it, or after it commits success; from the client's side these look alike. So a timeout is not a confirmed decline: returning PaymentResult.declined(...) claims "we know the payment did not succeed" when all we know is "we do not know". Nor is it a confirmed success: marking the Order PAID because the provider "probably processed it" could leave a PAID Order for a customer who was never charged. After an unresolved timeout, the Order is not marked PAID until the workflow has sufficient evidence.

This is harder than a database timeout. A local transaction that fails before commit leaves a controlled recovery model, but our transaction manager does not control the provider's persistence, and the network can fail after the external side effect happened. No local mechanism removes that outcome ambiguity.

Two Failure Types

The previous lesson's two failure types carry exactly the distinction the workflow needs: could the provider have charged the customer, or definitely not? PaymentProviderUnavailable means the provider did not process the request; PaymentOutcomeUnknown covers every case where it may have acted. Retry, reconciliation, the customer response and alerting treat the two differently; we need neither twenty subclasses nor a generic ExternalServiceException for every future integration.

Classifying Technical Failures

Each technical failure lands in one of the two types according to what it proves:

• Connection refused, unknown host. Nothing was delivered: PaymentProviderUnavailable. A DNS failure points to a bad provider URL or a deployment misconfiguration that operators need to see. A TLS handshake error is equally a system failure, and never "fixed" by disabling certificate validation.
• Provider 5xx. Provider-side technical trouble, never a decline. It does not prove that nothing happened either: the provider, or a gateway in front of it, can fail after booking the charge. Outcome unknown, as the previous lesson's client already treats it.
• Provider 4xx. An invalid request, an authentication problem or a business rejection, depending on the API; "all 4xx are declines" is no rule. A 401 for our invalid API credential is our configuration failing, and "required field missing" in a request we generated is an integration bug; a decline would hide either incident. Only statuses the contract documents as declines become PaymentResult.declined(...); the rest say the provider refused the request: PaymentProviderUnavailable.
• Malformed response. HTTP 200 with invalid JSON, a missing field or an unexpected schema: the provider received our request, and we cannot know from an unreadable answer whether the payment succeeded. Outcome unknown.
• Unknown status. The provider documented approved and declined and suddenly returns review. We must not silently choose either; until the contract is understood it is an unexpected response with an unknown outcome, which also makes contract changes visible.

The last two follow one rule for financial effects: success requires positive evidence. "Anything not explicitly failed is a success" is unsafe, and so is "anything not explicitly successful is a decline". This is failing closed. The previous lesson's client already follows it, with one gap it admitted: a refused connection also lands on "outcome unknown". Closing that gap needs a closer look at timeouts.

Timeouts Must Be Bounded and Configured

External calls must not wait indefinitely: a request thread blocked on a silent provider is lost capacity, and under load this exhausts threads and connections. HTTP clients distinguish two waits. The connect timeout bounds establishing the TCP (and TLS) connection; when it expires, nothing was sent. The read timeout bounds waiting for the response once the request is on its way; when it expires, the provider may have processed the request. ProviderPaymentProperties held a single timeout, used as the read timeout; because the two waits mean different things, we now configure them separately and keep the fail-fast checks:

package com.tarkev.ordermanagement.payment.integration.provider;

import java.net.URI;
import java.time.Duration;
import java.util.Objects;
import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties(prefix = "payment.provider")
public record ProviderPaymentProperties(
        URI baseUrl, Duration connectTimeout, Duration readTimeout) {

    public ProviderPaymentProperties {
        Objects.requireNonNull(baseUrl, "payment.provider.base-url is required");
        requirePositive(connectTimeout, "connect-timeout");
        requirePositive(readTimeout, "read-timeout");
        if (connectTimeout.compareTo(readTimeout) >= 0) {
            throw new IllegalArgumentException(
                    "payment.provider.connect-timeout must be shorter than read-timeout");
        }
    }

    private static void requirePositive(Duration timeout, String name) {
        Objects.requireNonNull(timeout, "payment.provider." + name + " is required");
        if (timeout.isZero() || timeout.isNegative()) {
            throw new IllegalArgumentException(
                    "payment.provider." + name + " must be positive");
        }
    }
}

The values belong to configuration, not to a constant buried inside PayOrderUseCase; production numbers come from an engineering decision and the provider's documented latency, which we do not invent here. The last check encodes a finding we are about to see: the connect timeout must be the shorter one. Locally:

payment:
  provider:
    base-url: http://localhost:9001
    connect-timeout: 1s
    read-timeout: 5s

Applying the Timeouts to RestClient

ProviderPaymentConfiguration builds the client's request factory through Boot's ClientHttpRequestFactoryBuilder. Spring Boot picks the HTTP library from the classpath in this order: Apache HttpClient 5, Jetty, Reactor Netty, the JDK's java.net.http.HttpClient, and the HttpURLConnection-based "simple" factory. spring-boot-starter-restclient brings none of the first three, so our application uses the JDK client unless we add a library or set spring.http.clients.imperative.factory. Only the settings line changes: it now passes both timeouts. The complete class:

package com.tarkev.ordermanagement.payment.integration.provider;

import com.tarkev.ordermanagement.payment.application.PaymentService;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.boot.http.client.ClientHttpRequestFactoryBuilder;
import org.springframework.boot.http.client.HttpClientSettings;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestClient;

@Configuration
@EnableConfigurationProperties(ProviderPaymentProperties.class)
public class ProviderPaymentConfiguration {

    @Bean
    ProviderPaymentClient providerPaymentClient(
            RestClient.Builder builder,
            ClientHttpRequestFactoryBuilder<?> requestFactoryBuilder,
            ProviderPaymentProperties properties) {
        HttpClientSettings settings = HttpClientSettings.defaults()
                .withTimeouts(properties.connectTimeout(), properties.readTimeout());
        RestClient restClient = builder
                .baseUrl(properties.baseUrl())
                .requestFactory(requestFactoryBuilder.build(settings))
                .build();
        return new ProviderPaymentClient(restClient);
    }

    @Bean
    PaymentService paymentService(ProviderPaymentClient client) {
        return new ProviderPaymentService(client);
    }
}

What a Timeout Looks Like in Spring Boot 4.1

Which exception does a timeout produce? We ran the client against a local stub server that waits three seconds before answering, with a read timeout of one second, and got ResourceAccessException: I/O error on POST request for "http://127.0.0.1:51616/payments": Request cancelled, caused by java.net.http.HttpTimeoutException. RestClient wraps I/O failures in ResourceAccessException; the cause comes from the HTTP library. Repeating the run per request factory and failure gives this picture:

Three findings shape our code. First, the types differ per library, so classification must search the cause chain, not match one class. Second, not every failure is a ResourceAccessException: with the simple factory the read timeout surfaced while reading the body, as a plain RestClientException, and with the JDK client a timeout that fired mid-body had a bare IOException: Request cancelled as cause. Third, with the JDK client the read timeout covers the whole exchange, connecting included: with a one-second read timeout and a two-second connect timeout, an unreachable host produced Request cancelled, indistinguishable from a read timeout. That is why the connect timeout must be the shorter one. The honest conclusion: the client can prove "never connected" only in some cases, and everything else must be treated as "the provider may have acted".

Recognizing a Connection Failure in ProviderPaymentClient

The rule stays deliberately asymmetric: only a 4xx and a proven connection failure become PaymentProviderUnavailable; every other broken exchange stays PaymentOutcomeUnknown. When unsure, we choose "outcome unknown", because the opposite mistake can charge a customer twice. The change to ProviderPaymentClient is the second catch block and a small helper that searches the cause chain:

package com.tarkev.ordermanagement.payment.integration.provider;

import com.tarkev.ordermanagement.payment.application.PaymentOutcomeUnknown;
import com.tarkev.ordermanagement.payment.application.PaymentProviderUnavailable;
import java.net.ConnectException;
import java.net.UnknownHostException;
import org.springframework.http.MediaType;
import org.springframework.web.client.RestClient;
import org.springframework.web.client.RestClientException;
import org.springframework.web.client.RestClientResponseException;

public class ProviderPaymentClient {

    private final RestClient restClient;

    public ProviderPaymentClient(RestClient restClient) {
        this.restClient = restClient;
    }

    ProviderPaymentResponse pay(ProviderPaymentRequest request) {
        ProviderPaymentResponse response;
        try {
            response = restClient.post()
                    .uri("/payments")
                    .contentType(MediaType.APPLICATION_JSON)
                    .body(request)
                    .retrieve()
                    .body(ProviderPaymentResponse.class);
        } catch (RestClientResponseException exception) {
            if (exception.getStatusCode().is4xxClientError()) {
                throw new PaymentProviderUnavailable(
                        "Provider answered " + exception.getStatusCode(), exception);
            }
            throw new PaymentOutcomeUnknown(
                    "Provider answered " + exception.getStatusCode(), exception);
        } catch (RestClientException exception) {
            if (causedBy(exception, ConnectException.class)
                    || causedBy(exception, UnknownHostException.class)) {
                throw new PaymentProviderUnavailable(
                        "Could not connect to provider", exception);
            }
            throw new PaymentOutcomeUnknown(
                    "No usable answer from provider", exception);
        }
        if (response == null) {
            throw new PaymentOutcomeUnknown("Provider sent no body", null);
        }
        return response;
    }

    private static boolean causedBy(Throwable exception, Class<?> type) {
        for (Throwable cause = exception; cause != null;
                cause = cause.getCause()) {
            if (type.isInstance(cause)) {
                return true;
            }
        }
        return false;
    }
}

The two cause checks cover a refused connection, an unknown host and the JDK client's connect timeout. The other libraries' connect timeouts and a malformed body land safely on "outcome unknown". Running ProviderPaymentService from the previous lesson with this client against the stub server, one answer at a time, produced the classification we want:

approved  -> PaymentResult[status=SUCCEEDED, providerPaymentId=p-1, declineReason=null]
declined  -> PaymentResult[status=DECLINED, providerPaymentId=null, declineReason=Declined by payment provider]
review    -> PaymentOutcomeUnknown: Unknown provider status: review
malformed -> PaymentOutcomeUnknown: No usable answer from provider
503       -> PaymentOutcomeUnknown: Provider answered 503 SERVICE_UNAVAILABLE
401       -> PaymentProviderUnavailable: Provider answered 401 UNAUTHORIZED
slow      -> PaymentOutcomeUnknown: No usable answer from provider
refused   -> PaymentProviderUnavailable: Could not connect to provider

Order State During Failure

After a decline, PaymentProviderUnavailable or PaymentOutcomeUnknown, the Order remains UNPAID, unless later reconciliation confirms success and the workflow safely applies it. A payment failure changes nothing else about the Order:

• It does not restore inventory. Inventory was consumed when the Order was created; restoration belongs only to a successful cancellation.
• It does not cancel the Order automatically. A timeout may hide a provider success, and "inventory restored, customer charged" is a serious inconsistency.
• It does not add states or fields to Order. Three attempts (declined, unavailable, timeout) had different outcomes, but the Order did not move through PAYMENT_FAILED, PAYMENT_ERROR or PAYMENT_TIMEOUT; those are attempt concepts, and so are lastPaymentError or providerUnavailable. A timeout may also tempt us to add PAYMENT_PENDING. Our requirements define no such state, and adding one would affect cancellation, payment eligibility, the API, the database CHECK constraint, the tests and the history semantics. If payment attempt state later needs persistence, it is modeled separately from the Order.

Retries Come Later

The classification also tells us when a retry might be safe. Ask: could the provider already have performed the action? If serialization failed locally or the connection was never established, the answer is "no"; once the request was transmitted, it is "maybe", and a blind retry after a lost success charges the customer twice. Spring Framework 7 makes retries very easy with @Retryable and RetryTemplate, and that ease is the danger. The next lesson works out when a retry is safe; the order is always: understand the failure, understand the provider guarantees, determine retry safety, and only then implement a retry policy.

An uncertain outcome also sharpens the Pay vs Cancel race: the Order still appears UNPAID, so the customer could cancel it and restore inventory while the provider may have charged them. Neither a row lock held across the call nor @Transactional around it helps; lessons 5 and 7 explain why and design the coordination.

Never convert a technical inability to determine the outcome into an expected business result just because the business result is easier to handle.

Customer-Facing Behaviour and Observability

A decline can later be reported through the API's stable error contract. After a timeout, the API must not say "your payment definitely failed", which invites a new attempt and duplicate-charge risk. The customer never receives HttpTimeoutException, HTTP 502, the provider host or provider codes. We deliberately do not invent a public PAYMENT_TIMEOUT code yet: a public code is a contract, and a provider status lookup might resolve a timeout before the customer sees an error. The Pay Order API designs that mapping later in this module.

Internally, operators need the OrderId, the correlation ID, the failure category and the provider reference if known; never the provider secret, the Authorization header or payment credentials. A decline is normal business behaviour, not an ERROR stack trace; a provider outage is significant. Production observability (module 11) counts declines, timeouts, unavailability and unknown outcomes separately; otherwise an outage looks like "more declined cards", and a declined card should not page anyone while a spike in timeouts might.

Tests

Each line of the classification run above is also a stub-server test; module 10 automates them. The contract may refine individual cases; the principle is that they are not one category, and that HTTP status codes are transport the adapter translates, not application concepts. At the PayOrderUseCase level, two more tests matter: with an UNPAID Order and a PaymentService that throws, the Order stays UNPAID and markPaid() is never called; with a declined result, the Order also stays UNPAID, which satisfies the Payment story's rule that a failed payment must not mark the Order as paid.

Practice Exercise 1: Prove the Asymmetric Rule

Solutions: lesson 9.12.

Write a stub-server test class M9ConnectionFailureTest for this lesson's ProviderPaymentClient, building the RestClient with withTimeouts(...): a 200 ms connect timeout and a 500 ms read timeout. Cover four cases and the classification each must produce:

• Connection refused: start the stub, note its port, stop it, then pay against that port. Expect PaymentProviderUnavailable with the message Could not connect to provider.
• 401: PaymentProviderUnavailable.
• 502: PaymentOutcomeUnknown.
• 200 with an approved body after two seconds: PaymentOutcomeUnknown, although the stub would have said approved.

The last case is the one to think about: in a real system, which of the four could have charged the customer?

Practice Exercise 2: Which Configuration Starts?

Predict whether the application starts with each payment.provider configuration, and if not, which message stops it:

1. connect-timeout: 1s, read-timeout: 5s
2. connect-timeout: 5s, read-timeout: 5s
3. connect-timeout: 500ms, read-timeout: 0s
4. connect-timeout: 1s, no read-timeout
5. connect-timeout: 2s, read-timeout: 1s

For the fifth, explain what would go wrong with the JDK client if the check did not exist.

Practice Exercise 3: Find the Problem

A teammate wants the client to recognize connect timeouts with every HTTP library and extends the cause check:

if (causedBy(exception, ConnectException.class)
        || causedBy(exception, UnknownHostException.class)
        || causedBy(exception, SocketTimeoutException.class)) {
    throw new PaymentProviderUnavailable(
            "Could not connect to provider", exception);
}

The reasoning in the pull request: "Apache's ConnectTimeoutException is a SocketTimeoutException." Use the lesson's table to find the problem. What can go wrong for a customer, and what would you do instead?

Knowledge Check

Question 1

Why does an expired connect timeout tell us more than an expired read timeout?

Question 2

The provider answers 200 with {"status": "review"}. Why is that neither a success nor a decline?

Question 3

Why should a decline not be logged as an ERROR, while a spike in timeouts may page someone?

Question 4

After PaymentOutcomeUnknown, why does the workflow not cancel the Order to give the Inventory back?

Question 5

Why does ProviderPaymentClient search the cause chain instead of catching one exception type such as HttpTimeoutException?

Knowledge Check Answers

Answer 1

When connecting fails, the request was never sent, so the provider cannot have acted. When reading times out, the request may have arrived and been processed; only the answer is missing.

Answer 2

Success requires positive evidence and a decline requires a confirmed refusal; an undocumented status is neither. Treating it as PaymentOutcomeUnknown fails closed and makes the contract change visible.

Answer 3

A decline is normal business behaviour that the customer can act on. Timeouts and unavailability point to an outage or a configuration problem, and counting them separately keeps an outage from looking like "more declined cards".

Answer 4

The provider may have charged the customer. Cancelling would restore Inventory and leave a charged customer with a CANCELLED Order, a worse inconsistency than an Order that stays UNPAID until the outcome is known.

Answer 5

Each HTTP library reports the same failure with a different exception, wrapped by RestClient in different ways. Only the cause chain reliably shows whether a connection was ever established.

Engineering Principle

An external-service error should be modeled according to what the application knows about the external side effect, not merely according to the local exception that occurred.

A timeout is an absence of timely confirmation, not proof of business failure, and because a retry can create a second external side effect, failure classification comes before retry policy.

Summary

• Only an explicit provider confirmation is a success; a confirmed decline is a normal PaymentResult; technical failures are never declines.
• A timeout proves neither success nor failure, and neither a local rollback nor @Transactional can resolve that ambiguity.
• PaymentProviderUnavailable (definitely not charged) and PaymentOutcomeUnknown (outcome unknown) carry the distinction the workflow needs.
• Spring Boot 4.1 applies the connect and read timeouts through HttpClientSettings and its ClientHttpRequestFactoryBuilder, which defaults to the JDK HttpClient; keep the connect timeout shorter.
• ProviderPaymentClient searches the library-specific cause chain and treats only a proven connection failure as "unavailable".
• The Order stays UNPAID through failed and uncertain attempts; failures never cancel it, restore inventory or add states to OrderStatus.
• Failure classification comes before any retry policy, because a blind retry can charge a customer twice.

The next lesson, Retry Safety and Idempotency, answers the question this one leaves open: when is it safe to try a failed or timed-out payment again? It also shows why Order.markPaid() rejecting a second transition does not prevent a customer from being charged twice.