Thinking Like a Backend Engineer

What Makes Backend Engineering Advanced?

LugemineEelvaade

Vaatad tasuta eelvaate õppetundi.

Building a backend application is relatively straightforward.

Building a backend system that remains reliable as traffic grows, dependencies fail, data is modified concurrently, requirements change, and many engineers work on it at the same time is much harder.

This is where Advanced Backend Engineering begins.

Advanced Backend Engineering is not simply about knowing more frameworks, libraries, infrastructure tools, or architectural patterns.

The real challenge is learning how to reason about systems:

How will the system behave? Where can it fail? How will it evolve? What trade-offs are acceptable?

What You Will Learn

By the end of this lesson, you should understand:

the difference between ordinary backend development and advanced backend engineering;
why working code does not necessarily mean a correct system;
how scale introduces new engineering problems;
why failure is a normal part of production systems;
how concurrency can break otherwise correct code;
why compatibility matters as systems evolve;
how team structure influences architecture;
why architecture is fundamentally about managing trade-offs.

1. Building a Feature Is Not the Same as Engineering a System

Suppose the business requirement is simple:

A customer should be able to create an order.

A basic implementation might look like this:

Client
  │
  ▼
OrderController
  │
  ▼
OrderService
  │
  ▼
OrderRepository
  │
  ▼
PostgreSQL

The client sends:

POST /orders

The application validates the request, creates an order, stores it in the database, and returns a response.

The feature works.

But production introduces a different set of questions:

What happens if the same request arrives twice?
What happens if the order is created but the client never receives the response?
What happens if the client retries?
Could two customers purchase the last available item at the same time?
What happens if the database becomes slow?
What happens if payment succeeds but inventory reservation fails?
What happens when traffic grows from 100 RPS to 10,000 RPS?
How do we know when the system becomes unhealthy?
Could a database schema change break an older application instance?

These are not primarily questions about controllers, annotations, or framework APIs.

They are system engineering questions.

A backend developer may be asked:

How do we implement this endpoint?

A backend engineer must also ask:

Under what conditions can this endpoint behave incorrectly?

That difference becomes increasingly important as systems move from development environments into production.

2. Working Code Does Not Necessarily Mean a Correct System

Consider this endpoint:

POST /orders

Request:

{
  "productId": "P123",
  "quantity": 1
}

The server creates an order:

{
  "orderId": "O456",
  "status": "CREATED"
}

Everything appears correct.

Now consider this sequence:

Client
   │
   │ POST /orders
   ▼
Server
   │
   ├── Order successfully created
   │
   ├── Response sent
   │
   X Network connection lost

From the server's perspective, the operation succeeded.

But the client never received the response.

The client cannot know whether:

The order was created

or

The request failed

So the client retries:

POST /orders

The server may create another order.

Both HTTP requests can be technically successful.

But the business result is incorrect:

Expected orders = 1

Actual orders = 2

This gives us an important principle:

Request-level correctness is not the same as business-level correctness.

Later in the course, we will study techniques such as idempotency for handling this type of problem.

For now, the goal is not to solve it.

The goal is to recognize that the problem exists.

3. Scale Creates Complexity

Suppose an application initially receives:

10 requests/second

Its architecture might be extremely simple:

Client
  │
  ▼
Application
  │
  ▼
PostgreSQL

This may work perfectly well.

Then traffic increases to:

1,000 requests/second

Later:

10,000 requests/second

Suddenly, decisions that barely mattered before can become significant.

For example:

slow SQL queries;
missing indexes;
too many database connections;
unnecessary network calls;
CPU-intensive processing;
synchronous operations;
large response payloads;
memory pressure;
hot database rows.

A common response might be to add more application instances:

                   Load Balancer
                        │
          ┌─────────────┼─────────────┐
          ▼             ▼             ▼
       API #1         API #2        API #3
          │             │             │
          └─────────────┼─────────────┘
                        ▼
                   PostgreSQL

The application layer now has more capacity.

But all three application instances may open additional connections to the same database.

The result might be:

Application bottleneck reduced

but

Database bottleneck introduced

The problem did not disappear.

It moved.

This leads to another principle:

Scaling one component does not automatically scale the entire system.

Backend engineers must understand the system end to end.

4. Failure Is a Normal Part of Production Systems

Production systems fail.

Not only during major outages.

Small failures happen continuously.

Examples include:

network timeouts;
slow database queries;
application restarts;
container termination;
DNS failures;
unavailable external APIs;
duplicate message delivery;
connection resets.

Failure is therefore not something that happens outside the normal operation of a distributed system.

It is part of normal operation.

Consider:

Order Service
      │
      ▼
Payment Service
      │
      ▼
Inventory Service

Suppose payment succeeds.

Then the inventory operation fails.

What should the system do?

Possible options include:

Refund the payment

Retry inventory reservation

Keep the order pending

Cancel the order

Send the case for manual review

Which option is correct?

There is no universal answer.

It depends on the business requirements.

For one system, immediately refunding the customer may be appropriate.

For another, retrying inventory reservation may be preferred.

Another business may require manual review.

This leads to an important principle:

Failure handling is part of architecture.

It is not an optional feature to add after the main implementation is finished.

5. Concurrency Can Break Correct Code

Consider this inventory table:

product_id | available_quantity
-----------|-------------------
P123       | 1

Only one item remains.

Two customers attempt to purchase it at almost exactly the same time.

Request A reads:

available_quantity = 1

Request B also reads:

available_quantity = 1

Both requests conclude that the item is available.

The application code might look perfectly reasonable:

if (product.getAvailableQuantity() > 0) {
    product.decreaseQuantity();
    repository.save(product);
}

If you execute this code once, it may behave correctly.

But if two requests execute it concurrently, the result may be wrong.

That is the concurrency problem.

Code that is correct when executed sequentially can become incorrect when executed concurrently.

Later in the course, we will study concepts such as:

transactions;
isolation levels;
optimistic locking;
pessimistic locking.

For now, the important point is simply to recognize concurrency as a first-class engineering concern.

6. Backend Systems Manage State

Backend systems ultimately manage state.

Examples include:

orders;
payments;
inventory;
account balances;
subscriptions;
identities;
permissions.

Incorrect state can be more dangerous than an obvious outage.

Consider:

Expected balance = €100

Actual balance = €150

The servers may still be running.

CPU usage may look completely normal.

Health checks may be green.

The API may still return:

200 OK

But the system is wrong.

This is why the backend engineer's responsibility is not limited to asking:

"Is the API running?"

The more important question is:

"Is the system's state correct?"

A healthy process does not necessarily mean a healthy business system.

7. Systems Become More Complex Over Time

Production systems are not static.

They evolve.

Suppose the first version of an API returns:

{
  "id": "123",
  "name": "Laptop"
}

A new requirement introduces a different field:

{
  "id": "123",
  "displayName": "Laptop"
}

At first glance, this appears to be a simple rename.

But consider what may already depend on the original field:

another backend service may still expect name;
an older mobile application may still expect name;
previously produced Kafka events may contain name;
an older background worker may still process the original schema.

Suddenly, a simple rename is no longer simple.

Production systems therefore require engineers to think about:

Backward compatibility
Schema evolution
Migration
Deprecation
Rollback
Deployment order

Architecture is not only about designing the desired final state.

It is also about:

Safely moving the system from today's state to tomorrow's state.

The migration path is part of the architecture.

8. Architecture Becomes More Important as Teams Grow

When one engineer owns an entire application, coordination is relatively simple.

Now imagine an organization with separate teams for:

Accounts
Catalog
Orders
Payments
Inventory
Shipping

Suppose the Orders team changes an event contract:

Orders change
     ↓
Inventory breaks

Or the Payments team changes an API:

Payments change
      ↓
Checkout breaks

If every team directly reads and modifies every other team's database, ownership becomes unclear.

If every change requires coordination across several teams, development becomes increasingly difficult.

Architecture helps define:

who owns which system;
who owns which data;
which interfaces are stable;
where dependencies are allowed;
which teams can deploy independently.

Architecture is therefore not only about software components.

It is also connected to how people and teams work together.

9. Advanced Engineers Do Not Start With Technology

A common question is:

Which database is best?

That question is usually too broad to be useful.

A better question is:

What does this workload require?

Before choosing a database, we should ask questions such as:

Do we need transactions?

How important is durability?

What are the access patterns?

How much data will we store?

Is the workload read-heavy or write-heavy?

Can the application tolerate stale data?

What latency target do we have?

How much operational complexity can the team support?

Only after understanding the requirements should we evaluate the technology.

The same principle applies to:

Kafka
Redis
Microservices
Kubernetes
Temporal
NoSQL databases

Do not begin with:

We should use Kafka.

Begin with:

What problem are we trying to solve?

Then evaluate whether Kafka is an appropriate solution.

Technology should follow requirements.

Not the other way around.

10. Every Solution Introduces New Problems

Suppose database reads are becoming slow.

You introduce Redis.

Before:

Application
     │
     ▼
PostgreSQL

After:

Application
     │
     ▼
   Redis
     │
 Cache miss
     ▼
PostgreSQL

Latency improves.

But now another problem appears.

Suppose PostgreSQL contains:

Price = €100

while Redis still contains:

Price = €90

Which value should the customer see?

The system improved:

Performance

but introduced another challenge:

Consistency

This pattern appears repeatedly in backend engineering:

Solving one problem often introduces a new trade-off.

Good engineering therefore requires evaluating both what a solution provides and what it costs.

11. Microservices Are Not Automatically Better

Microservices can provide useful properties such as:

independent deployment;
clearer team ownership;
independent scaling;
domain separation.

But those benefits come with costs:

network failures;
distributed transactions;
additional operational overhead;
more complicated observability;
deployment complexity;
cross-service debugging.

Therefore:

Using microservices does not automatically make an architecture advanced.

In many situations, a well-designed modular monolith may be the better engineering decision.

Advanced engineering does not mean choosing the most complex architecture.

It means choosing the architecture that best fits the constraints.

12. Architecture Is Trade-Off Management

Suppose an engineer says:

We should use Kafka because Kafka is scalable.

That is weak architectural reasoning.

A stronger explanation would be:

We require durable event retention, replay, partition-level ordering, and multiple independent consumers. Kafka provides those properties, but it also introduces additional infrastructure and operational complexity.

The second explanation considers both:

Benefits
+
Costs

That is architectural thinking.

Instead of asking:

Is Kafka good?

Ask:

What problem does Kafka solve for us, and what complexity do we accept in return?

The same reasoning applies to almost every major architectural decision.

13. The Definition of "Done" Changes in Production

For a simple application, "done" might mean:

✓ Code works
✓ Tests pass
✓ API responds correctly

For a production system, the definition is larger:

✓ Functional requirements work

✓ Concurrent operations remain correct

✓ Retries are safe

✓ Failure behavior is understood

✓ Useful logs exist

✓ Important metrics exist

✓ Deployment is safe

✓ Rollback is possible

✓ Database migrations remain compatible

✓ Capacity limits are understood

Production readiness is therefore not simply a property of application code.

It is a property of the entire system.

14. A Backend Engineer's Mental Checklist

Throughout this course, we will repeatedly use a small set of questions when reasoning about backend systems.

Correctness

Can this operation create an invalid business state?

Failure

What happens when this dependency fails?

Concurrency

What happens when this operation executes simultaneously?

Scale

What happens if traffic increases by 100×?

Data

Where is the source of truth?

Consistency

How stale can this data safely become?

Recovery

Can the system recover from partial failure?

Observability

How would we know that this system is broken?

Evolution

Could this change break an existing consumer?

Operations

Could another engineer understand what is happening during an incident?

Developing the habit of asking these questions is one of the foundations of advanced backend engineering.

Practical Exercise — The Simple Order System

Consider this architecture:

Client
  │
  ▼
Order API
  │
  ▼
PostgreSQL

The endpoint is:

POST /orders

Request:

{
  "productId": "P123",
  "quantity": 1
}

The application performs four operations:

1. Check available inventory
2. Create the order
3. Reduce inventory
4. Return the order

Assume the current traffic is:

100 requests/second

Now analyze the following scenarios.

Scenario 1 — Concurrent Purchase

Two customers attempt to purchase the final item at the same time.

Ask yourself:

What could go wrong?

Do not design the solution yet.

First, identify the engineering problem.

Scenario 2 — Client Retry

The server successfully creates an order.

Before the response reaches the client, the network connection disappears.

The client retries the request.

Ask:

What incorrect business state could be created?

Again, focus on identifying the problem rather than selecting a technology.

Scenario 3 — Slow Database

Database queries normally complete in:

20 ms

They suddenly begin taking:

5 seconds

Consider:

What happens to incoming requests?
What happens to application resources?
How could this affect the user experience?
Could this problem spread to other parts of the system?

Scenario 4 — Traffic Spike

Traffic increases from:

100 RPS

to:

10,000 RPS

Which components might become bottlenecks?

Again, do not immediately jump to technology.

First identify where pressure could appear.

Scenario 5 — Introducing Redis

Redis is added to reduce database reads.

Now ask:

What new consistency problems might appear?

Think about what happens when the cached representation and the database representation no longer contain the same value.

Exercise Format

For each scenario, answer three questions:

1. What can fail or behave incorrectly?

2. What could the business impact be?

3. What information would you need before choosing a solution?

The purpose of this exercise is not to select technology.

The purpose is to develop the habit of:

Understanding the problem before selecting the solution.

Key Takeaways

The most important ideas from this lesson are:

1. Working Code Is Not Enough

A production system must remain correct under failure, retries, concurrency, and change.

2. Scale Changes Assumptions

An architecture that works well at 100 RPS may behave very differently at 100,000 RPS.

3. Failure Is Normal

Production systems should be designed with failure in mind rather than assuming every dependency will always work.

4. Concurrency Changes Correctness

Code that works correctly when executed sequentially can become incorrect when executed concurrently.

5. Systems Evolve

Backward compatibility, migration, and safe evolution are fundamental engineering concerns.

6. Architecture Is Connected to Team Structure

Ownership, boundaries, and dependencies affect both software and organizations.

7. Every Solution Has a Cost

Redis, Kafka, microservices, and other architectural choices solve problems while introducing new forms of complexity.

8. Architecture Is About Trade-Offs

The important question is not:

Which technology is best?

The better question is:

Which trade-offs are appropriate for the requirements and constraints of this system?

Next Lesson

Lesson 2 — Understanding the Request Lifecycle

In the next lesson, we will follow a request through a backend system:

Client
  ↓
Load Balancer
  ↓
Application
  ↓
Cache
  ↓
Database
  ↓
External Dependencies

We will examine:

where latency is introduced;
where failures can occur;
where resources are consumed;
and why understanding the complete request lifecycle is essential before attempting to optimize a backend system.