Building an API is one thing.

Building a backend system that remains correct, reliable, observable, and maintainable as traffic grows, dependencies fail, data changes concurrently, and multiple engineering teams work on it is a very different challenge.

**Advanced Backend Engineering** is designed for backend developers who already understand application development and want to learn how production-grade backend systems are engineered.

Instead of focusing primarily on framework syntax, this course focuses on the engineering decisions behind reliable systems.

You will learn how to reason about architecture, service boundaries, databases, caching, asynchronous communication, distributed transactions, concurrency, failure recovery, performance, observability, and production operations.

Using **Java and Spring Boot** as the primary implementation stack, you will progressively design and build a distributed commerce platform containing services such as Orders, Inventory, Payments, Catalog, and Notifications.

Along the way, you will work with technologies and concepts including:

* Java and Spring Boot
* PostgreSQL
* Redis
* Kafka
* Debezium and Change Data Capture
* Temporal
* Docker
* Kubernetes
* OpenTelemetry
* Prometheus
* Grafana

But the goal of the course is not simply to teach these technologies.

The goal is to teach you **when to use them, why to use them, what problems they solve, and what new trade-offs they introduce**.

You will explore real production engineering problems such as duplicate requests, concurrent updates, database contention, stale caches, failed dependencies, message redelivery, partial transactions, consumer lag, cascading failures, unsafe schema migrations, and production incidents.

By the end of the course, you should be able to approach backend systems not simply as an application developer, but as an engineer capable of owning systems in production.

## What You Will Learn

By completing this course, you will learn how to:

* design backend architectures around real system requirements;
* identify appropriate service and domain boundaries;
* design APIs that remain safe under retries and concurrency;
* model relational data around access patterns;
* understand database transactions, isolation, locking, and contention;
* use caching effectively while managing consistency;
* build event-driven systems with Kafka;
* design idempotent producers and consumers;
* solve the database-and-message dual-write problem;
* implement transactional outbox and inbox patterns;
* use Change Data Capture with Debezium;
* design distributed transactions using sagas;
* understand orchestration and choreography;
* reason about race conditions and distributed coordination;
* implement retries, timeouts, circuit breakers, and backpressure;
* design systems for partial failure;
* build production observability using logs, metrics, and traces;
* define SLIs, SLOs, and error budgets;
* investigate latency and performance bottlenecks;
* estimate capacity and scale services;
* containerize and operate backend services with Kubernetes;
* perform safe database and API migrations;
* design zero-downtime deployment strategies;
* investigate and respond to production incidents;
* write technical RFCs and Architecture Decision Records;
* communicate architectural trade-offs clearly.

## Who This Course Is For

This course is designed for:

* backend developers who already know how to build REST APIs;
* Java and Spring Boot developers moving beyond CRUD applications;
* engineers preparing for senior backend engineering responsibilities;
* engineers working with distributed systems;
* developers who want stronger system design skills;
* engineers who want to understand production reliability and operations;
* experienced developers preparing for backend system-design interviews.

This is **not a beginner programming course**.

Learners should already be comfortable writing backend applications.

## Prerequisites

You should already understand:

* Java fundamentals;
* object-oriented programming;
* Spring Boot;
* REST APIs;
* HTTP fundamentals;
* relational databases;
* SQL;
* basic Git;
* basic Docker concepts.

Experience with PostgreSQL is recommended.

Kafka, Redis, Kubernetes, Temporal, Prometheus, and distributed systems knowledge are **not required**. They will be introduced as part of the course.

Building a backend application is relatively straightforward.

Building a backend system that remains reliable as traffic grows, dependencies fail, data is modified concurrently, requirements change, and many engineers work on it at the same time is much harder.

This is where **Advanced Backend Engineering** begins.

Advanced Backend Engineering is not simply about knowing more frameworks, libraries, infrastructure tools, or architectural patterns.

The real challenge is learning how to reason about systems:

> **How will the system behave? Where can it fail? How will it evolve? What trade-offs are acceptable?**

---

# What You Will Learn

By the end of this lesson, you should understand:

- the difference between ordinary backend development and advanced backend engineering;
- why working code does not necessarily mean a correct system;
- how scale introduces new engineering problems;
- why failure is a normal part of production systems;
- how concurrency can break otherwise correct code;
- why compatibility matters as systems evolve;
- how team structure influences architecture;
- why architecture is fundamentally about managing trade-offs.

---

# 1. Building a Feature Is Not the Same as Engineering a System

Suppose the business requirement is simple:

> A customer should be able to create an order.

A basic implementation might look like this:

```text
Client
  │
  ▼
OrderController
  │
  ▼
OrderService
  │
  ▼
OrderRepository
  │
  ▼
PostgreSQL
```

The client sends:

```http
POST /orders
```

The application validates the request, creates an order, stores it in the database, and returns a response.

The feature works.

But production introduces a different set of questions:

- What happens if the same request arrives twice?
- What happens if the order is created but the client never receives the response?
- What happens if the client retries?
- Could two customers purchase the last available item at the same time?
- What happens if the database becomes slow?
- What happens if payment succeeds but inventory reservation fails?
- What happens when traffic grows from 100 RPS to 10,000 RPS?
- How do we know when the system becomes unhealthy?
- Could a database schema change break an older application instance?

These are not primarily questions about controllers, annotations, or framework APIs.

They are **system engineering questions**.

A backend developer may be asked:

> How do we implement this endpoint?

A backend engineer must also ask:

> Under what conditions can this endpoint behave incorrectly?

That difference becomes increasingly important as systems move from development environments into production.

---

# 2. Working Code Does Not Necessarily Mean a Correct System

Consider this endpoint:

```http
POST /orders
```

Request:

```json
{
  "productId": "P123",
  "quantity": 1
}
```

The server creates an order:

```json
{
  "orderId": "O456",
  "status": "CREATED"
}
```

Everything appears correct.

Now consider this sequence:

```text
Client
   │
   │ POST /orders
   ▼
Server
   │
   ├── Order successfully created
   │
   ├── Response sent
   │
   X Network connection lost
```

From the server's perspective, the operation succeeded.

But the client never received the response.

The client cannot know whether:

```text
The order was created

or

The request failed
```

So the client retries:

```http
POST /orders
```

The server may create another order.

Both HTTP requests can be technically successful.

But the business result is incorrect:

```text
Expected orders = 1

Actual orders = 2
```

This gives us an important principle:

> **Request-level correctness is not the same as business-level correctness.**

Later in the course, we will study techniques such as **idempotency** for handling this type of problem.

For now, the goal is not to solve it.

The goal is to recognize that the problem exists.

---

# 3. Scale Creates Complexity

Suppose an application initially receives:

```text
10 requests/second
```

Its architecture might be extremely simple:

```text
Client
  │
  ▼
Application
  │
  ▼
PostgreSQL
```

This may work perfectly well.

Then traffic increases to:

```text
1,000 requests/second
```

Later:

```text
10,000 requests/second
```

Suddenly, decisions that barely mattered before can become significant.

For example:

- slow SQL queries;
- missing indexes;
- too many database connections;
- unnecessary network calls;
- CPU-intensive processing;
- synchronous operations;
- large response payloads;
- memory pressure;
- hot database rows.

A common response might be to add more application instances:

```text
                   Load Balancer
                        │
          ┌─────────────┼─────────────┐
          ▼             ▼             ▼
       API #1         API #2        API #3
          │             │             │
          └─────────────┼─────────────┘
                        ▼
                   PostgreSQL
```

The application layer now has more capacity.

But all three application instances may open additional connections to the same database.

The result might be:

```text
Application bottleneck reduced

but

Database bottleneck introduced
```

The problem did not disappear.

It moved.

This leads to another principle:

> **Scaling one component does not automatically scale the entire system.**

Backend engineers must understand the system end to end.

---

# 4. Failure Is a Normal Part of Production Systems

Production systems fail.

Not only during major outages.

Small failures happen continuously.

Examples include:

- network timeouts;
- slow database queries;
- application restarts;
- container termination;
- DNS failures;
- unavailable external APIs;
- duplicate message delivery;
- connection resets.

Failure is therefore not something that happens outside the normal operation of a distributed system.

It is part of normal operation.

Consider:

```text
Order Service
      │
      ▼
Payment Service
      │
      ▼
Inventory Service
```

Suppose payment succeeds.

Then the inventory operation fails.

What should the system do?

Possible options include:

```text
Refund the payment

Retry inventory reservation

Keep the order pending

Cancel the order

Send the case for manual review
```

Which option is correct?

There is no universal answer.

It depends on the business requirements.

For one system, immediately refunding the customer may be appropriate.

For another, retrying inventory reservation may be preferred.

Another business may require manual review.

This leads to an important principle:

> **Failure handling is part of architecture.**

It is not an optional feature to add after the main implementation is finished.

---

# 5. Concurrency Can Break Correct Code

Consider this inventory table:

```text
product_id | available_quantity
-----------|-------------------
P123       | 1
```

Only one item remains.

Two customers attempt to purchase it at almost exactly the same time.

Request A reads:

```text
available_quantity = 1
```

Request B also reads:

```text
available_quantity = 1
```

Both requests conclude that the item is available.

The application code might look perfectly reasonable:

```java
if (product.getAvailableQuantity() > 0) {
    product.decreaseQuantity();
    repository.save(product);
}
```

If you execute this code once, it may behave correctly.

But if two requests execute it concurrently, the result may be wrong.

That is the concurrency problem.

> **Code that is correct when executed sequentially can become incorrect when executed concurrently.**

Later in the course, we will study concepts such as:

- transactions;
- isolation levels;
- optimistic locking;
- pessimistic locking.

For now, the important point is simply to recognize concurrency as a first-class engineering concern.

---

# 6. Backend Systems Manage State

Backend systems ultimately manage state.

Examples include:

- orders;
- payments;
- inventory;
- account balances;
- subscriptions;
- identities;
- permissions.

Incorrect state can be more dangerous than an obvious outage.

Consider:

```text
Expected balance = €100

Actual balance = €150
```

The servers may still be running.

CPU usage may look completely normal.

Health checks may be green.

The API may still return:

```http
200 OK
```

But the system is wrong.

This is why the backend engineer's responsibility is not limited to asking:

```text
"Is the API running?"
```

The more important question is:

```text
"Is the system's state correct?"
```

A healthy process does not necessarily mean a healthy business system.

---

# 7. Systems Become More Complex Over Time

Production systems are not static.

They evolve.

Suppose the first version of an API returns:

```json
{
  "id": "123",
  "name": "Laptop"
}
```

A new requirement introduces a different field:

```json
{
  "id": "123",
  "displayName": "Laptop"
}
```

At first glance, this appears to be a simple rename.

But consider what may already depend on the original field:

- another backend service may still expect `name`;
- an older mobile application may still expect `name`;
- previously produced Kafka events may contain `name`;
- an older background worker may still process the original schema.

Suddenly, a simple rename is no longer simple.

Production systems therefore require engineers to think about:

```text
Backward compatibility
Schema evolution
Migration
Deprecation
Rollback
Deployment order
```

Architecture is not only about designing the desired final state.

It is also about:

> **Safely moving the system from today's state to tomorrow's state.**

The migration path is part of the architecture.

---

# 8. Architecture Becomes More Important as Teams Grow

When one engineer owns an entire application, coordination is relatively simple.

Now imagine an organization with separate teams for:

```text
Accounts
Catalog
Orders
Payments
Inventory
Shipping
```

Suppose the Orders team changes an event contract:

```text
Orders change
     ↓
Inventory breaks
```

Or the Payments team changes an API:

```text
Payments change
      ↓
Checkout breaks
```

If every team directly reads and modifies every other team's database, ownership becomes unclear.

If every change requires coordination across several teams, development becomes increasingly difficult.

Architecture helps define:

- who owns which system;
- who owns which data;
- which interfaces are stable;
- where dependencies are allowed;
- which teams can deploy independently.

Architecture is therefore not only about software components.

It is also connected to how people and teams work together.

---

# 9. Advanced Engineers Do Not Start With Technology

A common question is:

> Which database is best?

That question is usually too broad to be useful.

A better question is:

> What does this workload require?

Before choosing a database, we should ask questions such as:

```text
Do we need transactions?

How important is durability?

What are the access patterns?

How much data will we store?

Is the workload read-heavy or write-heavy?

Can the application tolerate stale data?

What latency target do we have?

How much operational complexity can the team support?
```

Only after understanding the requirements should we evaluate the technology.

The same principle applies to:

```text
Kafka
Redis
Microservices
Kubernetes
Temporal
NoSQL databases
```

Do not begin with:

> We should use Kafka.

Begin with:

> What problem are we trying to solve?

Then evaluate whether Kafka is an appropriate solution.

> **Technology should follow requirements.**

Not the other way around.

---

# 10. Every Solution Introduces New Problems

Suppose database reads are becoming slow.

You introduce Redis.

Before:

```text
Application
     │
     ▼
PostgreSQL
```

After:

```text
Application
     │
     ▼
   Redis
     │
 Cache miss
     ▼
PostgreSQL
```

Latency improves.

But now another problem appears.

Suppose PostgreSQL contains:

```text
Price = €100
```

while Redis still contains:

```text
Price = €90
```

Which value should the customer see?

The system improved:

```text
Performance
```

but introduced another challenge:

```text
Consistency
```

This pattern appears repeatedly in backend engineering:

> **Solving one problem often introduces a new trade-off.**

Good engineering therefore requires evaluating both what a solution provides and what it costs.

---

# 11. Microservices Are Not Automatically Better

Microservices can provide useful properties such as:

- independent deployment;
- clearer team ownership;
- independent scaling;
- domain separation.

But those benefits come with costs:

- network failures;
- distributed transactions;
- additional operational overhead;
- more complicated observability;
- deployment complexity;
- cross-service debugging.

Therefore:

> **Using microservices does not automatically make an architecture advanced.**

In many situations, a well-designed modular monolith may be the better engineering decision.

Advanced engineering does not mean choosing the most complex architecture.

It means choosing the architecture that best fits the constraints.

---

# 12. Architecture Is Trade-Off Management

Suppose an engineer says:

> We should use Kafka because Kafka is scalable.

That is weak architectural reasoning.

A stronger explanation would be:

> We require durable event retention, replay, partition-level ordering, and multiple independent consumers. Kafka provides those properties, but it also introduces additional infrastructure and operational complexity.

The second explanation considers both:

```text
Benefits
+
Costs
```

That is architectural thinking.

Instead of asking:

> Is Kafka good?

Ask:

> What problem does Kafka solve for us, and what complexity do we accept in return?

The same reasoning applies to almost every major architectural decision.

---

# 13. The Definition of "Done" Changes in Production

For a simple application, "done" might mean:

```text
✓ Code works
✓ Tests pass
✓ API responds correctly
```

For a production system, the definition is larger:

```text
✓ Functional requirements work

✓ Concurrent operations remain correct

✓ Retries are safe

✓ Failure behavior is understood

✓ Useful logs exist

✓ Important metrics exist

✓ Deployment is safe

✓ Rollback is possible

✓ Database migrations remain compatible

✓ Capacity limits are understood
```

Production readiness is therefore not simply a property of application code.

It is a property of the entire system.

---

# 14. A Backend Engineer's Mental Checklist

Throughout this course, we will repeatedly use a small set of questions when reasoning about backend systems.

## Correctness

```text
Can this operation create an invalid business state?
```

## Failure

```text
What happens when this dependency fails?
```

## Concurrency

```text
What happens when this operation executes simultaneously?
```

## Scale

```text
What happens if traffic increases by 100×?
```

## Data

```text
Where is the source of truth?
```

## Consistency

```text
How stale can this data safely become?
```

## Recovery

```text
Can the system recover from partial failure?
```

## Observability

```text
How would we know that this system is broken?
```

## Evolution

```text
Could this change break an existing consumer?
```

## Operations

```text
Could another engineer understand what is happening during an incident?
```

Developing the habit of asking these questions is one of the foundations of advanced backend engineering.

---

# Practical Exercise — The Simple Order System

Consider this architecture:

```text
Client
  │
  ▼
Order API
  │
  ▼
PostgreSQL
```

The endpoint is:

```http
POST /orders
```

Request:

```json
{
  "productId": "P123",
  "quantity": 1
}
```

The application performs four operations:

```text
1. Check available inventory
2. Create the order
3. Reduce inventory
4. Return the order
```

Assume the current traffic is:

```text
100 requests/second
```

Now analyze the following scenarios.

---

## Scenario 1 — Concurrent Purchase

Two customers attempt to purchase the final item at the same time.

Ask yourself:

```text
What could go wrong?
```

Do not design the solution yet.

First, identify the engineering problem.

---

## Scenario 2 — Client Retry

The server successfully creates an order.

Before the response reaches the client, the network connection disappears.

The client retries the request.

Ask:

```text
What incorrect business state could be created?
```

Again, focus on identifying the problem rather than selecting a technology.

---

## Scenario 3 — Slow Database

Database queries normally complete in:

```text
20 ms
```

They suddenly begin taking:

```text
5 seconds
```

Consider:

- What happens to incoming requests?
- What happens to application resources?
- How could this affect the user experience?
- Could this problem spread to other parts of the system?

---

## Scenario 4 — Traffic Spike

Traffic increases from:

```text
100 RPS
```

to:

```text
10,000 RPS
```

Which components might become bottlenecks?

Again, do not immediately jump to technology.

First identify where pressure could appear.

---

## Scenario 5 — Introducing Redis

Redis is added to reduce database reads.

Now ask:

```text
What new consistency problems might appear?
```

Think about what happens when the cached representation and the database representation no longer contain the same value.

---

# Exercise Format

For each scenario, answer three questions:

```text
1. What can fail or behave incorrectly?

2. What could the business impact be?

3. What information would you need before choosing a solution?
```

The purpose of this exercise is not to select technology.

The purpose is to develop the habit of:

> **Understanding the problem before selecting the solution.**

---

# Key Takeaways

The most important ideas from this lesson are:

## 1. Working Code Is Not Enough

A production system must remain correct under failure, retries, concurrency, and change.

## 2. Scale Changes Assumptions

An architecture that works well at 100 RPS may behave very differently at 100,000 RPS.

## 3. Failure Is Normal

Production systems should be designed with failure in mind rather than assuming every dependency will always work.

## 4. Concurrency Changes Correctness

Code that works correctly when executed sequentially can become incorrect when executed concurrently.

## 5. Systems Evolve

Backward compatibility, migration, and safe evolution are fundamental engineering concerns.

## 6. Architecture Is Connected to Team Structure

Ownership, boundaries, and dependencies affect both software and organizations.

## 7. Every Solution Has a Cost

Redis, Kafka, microservices, and other architectural choices solve problems while introducing new forms of complexity.

## 8. Architecture Is About Trade-Offs

The important question is not:

> Which technology is best?

The better question is:

> **Which trade-offs are appropriate for the requirements and constraints of this system?**

---

# Next Lesson

## Lesson 2 — Understanding the Request Lifecycle

In the next lesson, we will follow a request through a backend system:

```text
Client
  ↓
Load Balancer
  ↓
Application
  ↓
Cache
  ↓
Database
  ↓
External Dependencies
```

We will examine:

- where latency is introduced;
- where failures can occur;
- where resources are consumed;
- and why understanding the complete request lifecycle is essential before attempting to optimize a backend system.

What Makes Backend Engineering Advanced?

Understanding the Request Lifecycle

Functional vs Non-Functional Requirements

Capacity Estimation

Thinking Like a Backend Engineer

Modular Monolith vs Microservices

Finding Service Boundaries

Domain-Driven Design

Coupling and Cohesion

Designing Service Architecture

Go beyond building APIs and learn how production backend systems are actually designed and operated.

This advanced course covers distributed systems, data consistency, Kafka, caching, resilience, observability, SLOs, performance engineering, Kubernetes, safe deployments, and architectural decision-making using Java and Spring Boot.

Design, build, scale, and operate reliable production backend systems.

Advanced Backend Engineering

Õppetunnile juurdepääsu kontrollimine...