How Senior Engineers Actually Think About System Design & Architecture | Full Course

Core idea

“Anyone can build apps”—system design is about handling scale, failures, and tradeoffs.

Building a basic app (frontend + backend + DB + deployment) is usually straightforward. The real difficulty starts when:

• traffic spikes
• the database/infra becomes a bottleneck
• latency appears
• failures occur

The proposed learning approach: start simple (one server + one DB) and then progressively “break” the system to understand why each architectural change is needed—along with its costs and tradeoffs.

---

Step-by-step scaling and architecture changes

1) Start: 1 server + 1 database

• Classic architecture: one app server runs the code; one database stores the data.
• This setup can handle surprisingly large loads (the example context suggests up to around “thousands of users” / about a thousand).

---

2) Traffic spike → vertical vs horizontal scaling

Example: ticket sales for a concert create massive simultaneous traffic, causing crashes like:

• slow requests
• timeouts
• queued requests piling up

Root cause: a single server can only process a limited amount of concurrent work, so requests queue and time out.

Two scaling approaches:

• Vertical scaling: upgrade to a bigger machine (more CPU/RAM).
• Horizontal scaling: run multiple identical app instances and distribute traffic.

---

3) Horizontal scaling → load balancer in front of multiple instances

• Multiple app instances are copies of the same application (not microservices by function).
• A load balancer routes each incoming request to one instance.
• Examples/tools mentioned:
  • NGINX as a classic load balancer
  • platforms like Vercel/Railway that may hide some load-balancing complexity

Tradeoff/cost:

• The load balancer becomes a single point of failure, so it needs redundancy.
• Instances must be interchangeable.

---

4) Multiple servers break login sessions → stateless servers + shared session storage

Problem: with multiple instances, a user may log in on server A but later requests go to server B, which doesn’t know the session.

Why it happens:

• login typically creates a session ID
• that session state is often stored in server memory (RAM)
• with load balancing, session state must be shared across servers

Solution:

• make servers stateless (don’t store per-user session in local memory)
• store session state in shared storage like Redis

Tradeoff/cost:

• every request now requires an extra lookup in shared storage
• the shared session store must be highly available (if it fails, logins fail)

---

5) App becomes slow again, but servers look healthy → DB bottlenecks

Scaling app servers doesn’t automatically scale the database. Two major DB issues:

(a) Connection limits → connection pooling

• Databases have limits on open connections.
• More app instances (and especially serverless) can cause connection storms.
• Solution: use connection pooling so servers reuse a small fixed number of DB connections.
  • Example: PgBouncer for Postgres (managed DBs may include pooling).

Symptom: “database connection errors” during traffic spikes.

(b) DB is CPU/work overloaded → read replicas

• Many real systems are read-heavy (many reads per write).
• Solution:
  • Primary database: handles writes
  • Read replicas: read-only copies that receive replicated updates
  • distribute reads across replicas

Tradeoff/cost:

• replication latency can cause temporarily stale reads
• some flows may require reading from the primary to avoid issues like “missing your own post”

---

6) Expensive repeated queries → caching

Problem: some computations repeat constantly (e.g., follower counts). Recounting millions of followers each time is wasteful.

Solution: cache computed results in a fast store (often Redis in RAM).

Typical flow:

1. check cache first
2. if missing, query the DB
3. write the result back to cache

Key tradeoff/cost:

• cached values can be temporarily inaccurate until refreshed/evicted

Requires deciding which data can be stale:

• follower count: slightly late is often acceptable
• account balance: should never be slightly wrong (so don’t cache)

Emphasis: caching decisions are a product/user decision, not just a coding decision.

---

7) Slow responses caused by external services → queues + background workers

Example: registration waits on tasks like sending confirmation emails.

Issue:

• the request thread blocks while calling third-party services
• latency becomes dependent on external systems

Solution:

• use a queue for deferred/non-critical work
• use a worker to process queued tasks in the background

Tooling mentioned:

• BullMQ running on Redis

Benefits:

• the user gets “account created” quickly
• email can happen later; if the email provider is down, the task waits

Tradeoffs/cost:

• added infrastructure (queue + worker management)
• tasks can fail → need retries/backoff
• “done” changes (promise vs actual completion time)

---

8) Data volume too large for one machine → sharding

Problem: read replicas assume the data fits on a single machine. If it doesn’t, you must distribute it across multiple databases (shards).

Mechanism:

• choose a sharding key that exists in every row (example: user_id)
• use a consistent rule (e.g., hash/modulo) to map rows to shards
• ensures the same user always maps to the same shard

Key benefit:

• single-user read/write typically goes to exactly one shard

Key tradeoff:

• “global” queries require inter-shard queries, such as:
  • counting all users
  • sorting across all records
  • retrieving data spanning partitions
• poor sharding can cause many queries to touch all shards

Migration pain:

• moving to sharding in production is extremely complex and risky
• example cited: Notion reportedly migrated to sharding with 200B+ rows while continuing to run

---

Overall “architecture chain” built in the video

1. 1 server + 1 database
2. add horizontal scaling: multiple app instances + load balancer
3. make apps stateless: sessions in Redis
4. fix DB spikes: connection pooling
5. fix DB overload: read replicas
6. fix repeated expensive reads: caching (Redis)
7. fix third-party latency: queue + workers (BullMQ/Redis)
8. fix DB storage size: sharding

---

Emphasis on “senior engineer thinking” / learning outcome

• The core skill is choosing the right components and their order, understanding tradeoffs, and knowing what you’re willing to make slightly inaccurate (especially caching).
• AI can generate implementation blocks, but can’t reliably make the best architectural/product decisions and cost judgments.
• Mentions an “Architect” tool to help walk through decisions in your own project.

---

Main speakers/sources

• Speaker: Adrian (introduced as “Hello everyone. I’m Adrian.”)
• Primary source: Adrian’s narration/teaching within the video:
  • “How Senior Engineers Actually Think About System Design & Architecture | Full Course”