Summary of "Highload 1. Сбор требований. Фронтенд-бэкенд"
Summary — key technical concepts, product/architecture guidance and practical patterns
Context
A lecture covered requirement gathering and basic high‑load architecture for a social‑network–style product (frontend/backend design). Emphasis was placed on how an architect should question managers, translate product needs into technical requirements, and choose simple architectural patterns that solve performance and scalability problems without heavy rewrites.
Requirements gathering — what to ask and why
Gather both functional and non‑functional information, plus data, load and business constraints:
- Functional requirements
- What users see and can do: pages, actions, user types, subscriptions, feeds.
- Content types and relationships: content tied to places, guides, etc.
- Non‑functional requirements
- Response times and latency goals.
- Uptime/SLA and tolerance for downtime.
- Acceptable functional degradation under load.
- Spike behavior and expected spike multipliers.
- Security and privacy (e.g., GDPR deletion rules).
- Data model questions
- Entity types and relationships: users, guides, places, articles, photos, videos.
- Data lifecycle: retention, deletion rules, physical vs logical deletion.
- Volumes: average and maximum sizes, growth projections, regional/replication constraints.
- Load and usage profile
- Baseline traffic and peak/spike multipliers.
- Geographic and temporal patterns.
- Client bandwidth characteristics (e.g., slow mobile users).
- Read/write ratios and concurrency expectations.
- Business constraints
- Budget, deadlines, and available team skills.
- How cost and developer availability affect technology choices.
Important: collect averages and maxima to properly size storage, bandwidth, memory and compute. Plan for future changes (new data types, worldwide expansion) to avoid short‑lived architectures.
Design principles and trade‑offs
- Plan for graceful functional degradation: relax freshness or hide non‑essential blocks under load (for example, extend page cache duration during spikes).
- Prefer simple, well‑known solutions first; many projects scale far on a single well‑configured machine before distributed complexity is necessary.
- Technology or language choice rarely dictates high‑level architecture — exceptions exist for specialized components (e.g., long‑lived keepalive servers may be easier in C++/Go).
- Expect to rewrite parts later as requirements and data evolve; design for growth and extensibility.
Three‑link (three‑layer) architecture pattern (core pattern)
A practical core pattern: Frontend → Backend → Storage.
Frontend (edge / reverse proxy / lightweight web server)
- Responsibilities: serve static assets, caching, buffering, TLS termination, basic routing and error pages, protect backends from slow clients and simple DoS.
- Example: Nginx.
- Benefits: serves static files quickly, maintains many concurrent connections cheaply, buffers slow uploads/downloads so backends aren’t blocked, provides caching and short‑term personalization caching.
Backend (application servers)
- Responsibilities: dynamic computations such as database queries, assembling feeds, rendering dynamic HTML/APIs.
- Resource considerations: memory and per‑process footprint, per‑request CPU/time, concurrency limits, and the number of backend instances required.
Storage (databases / object storage / caches)
- Responsibilities: persistent data — databases, distributed indices, object stores for images/videos, caches.
- Consider sharding or partitioning to split large volumes across multiple DB instances.
- Database choice is orthogonal to the sharding/partitioning pattern.
Operational patterns and tuning
- Caching: server‑side page/fragment caching, static asset caching, and using frontends to cache frequently requested content to drastically reduce backend load.
- Buffering at the frontend: absorb slow uploads/downloads so backends handle only fully‑received requests (critical for large uploads and poor client connections).
- Load balancing: distribute dynamic requests across backend instances and across DB shards when applicable.
- Progressive image formats and delivery optimizations to improve perceived performance.
- Error handling and fallback presentation: configure frontends to show friendly placeholders or omit failed blocks (functional degradation).
- Capacity planning: compute storage totals (counts × average/max sizes), bandwidth & transfer limits, memory per process, and concurrency ceilings.
- Security and isolation: frontends hide backend and DB endpoints from the public internet; protect databases from direct external connections.
Sizing and planning advice
- Always collect average and extreme (max) values to design for normal and pathological cases.
- Consider lifecycle rules and GDPR requirements (physical deletion vs marking deleted).
- Model growth scenarios (e.g., 100 places → 10,000 places) and make scaling incremental: start small and plan to add servers/shards as needed.
- For tight deadlines or budgets, favor patterns that increase capacity with minimal code changes (add an Nginx frontend, enable caches, add replicas) before planning a rewrite.
Course/process notes
- Curriculum outline: study ~20 architectural patterns (“architectural cartridges”), learn a simple design algorithm (5–6 steps), then apply patterns to projects with varying complexity.
- The three‑link pattern (frontend‑backend‑storage) is the first core example; other simple, low‑cost patterns follow.
Concrete examples referenced
- Functional degradation: Yandex main page blocks (weather/news) can be omitted without breaking the page.
- Photo storage and distribution issues in early social networks (VKontakte example).
- Handling large spikes (news/terror events) by increasing cache durations to survive 20× traffic surges.
- YouTube’s upload/processing workflow as an example of asynchronous processing and delayed availability.
- Real‑world systems and people referenced: Nginx (Igor Sysoev), VKontakte (Pavel Durov), Yandex, YouTube, LiveJournal, Facebook.
Actionable checklist for architects interviewing product managers
- Capture functional user journeys and per‑page expectations (what must be shown vs optional).
- Enumerate all entity types and relationships.
- Collect quantitative metrics: counts, average/max sizes, retention periods, read/write ratios, concurrency, regional constraints.
- Define non‑functional goals: latency, SLA/uptime, acceptable degradation levels, spike tolerance.
- Identify business limits: budget, time‑to‑market, available team skills.
- Choose simple patterns first (frontend caching/proxy, buffering, basic load balancing) and iterate.
Main speakers / sources referenced
- Unnamed lecturer / system architect (Highload lecture series speaker).
- Referenced individuals and systems: Pavel Durov (VKontakte), Igor Sysoev (Nginx), Yandex, VKontakte, YouTube, LiveJournal, Facebook.
Category
Technology
Share this summary
Is the summary off?
If you think the summary is inaccurate, you can reprocess it with the latest model.
Preparing reprocess...