Video summary

Все, что нужно знать про мониторинг

Main summary

Key takeaways

Technology

Summary of technological concepts & product/tool features

1) Why monitoring matters

  • Any system will eventually fail; the key question is whether monitoring lets you detect failures before users/customers notice them.
  • Monitoring evolves with infrastructure growth: from simple server checks to monitoring thousands of microservices.
  • “Monitoring” includes far more than dashboards/graphs—modern monitoring also covers application behavior and business outcomes.

2) Evolution of monitoring (by era)

1980s–1990s: physical servers + basic network polling

Monitoring focused on whether a server is alive, disk is full, and CPU is overheated.

Tools/protocols mentioned:

  • SNMP (network polling of routers/switches): collects counters like traffic, errors, and port status; still widely used.
  • MRTG: builds traffic graphs from SNMP data.
  • RRDTool (evolved from MRTG): time-series storage/graphing approach.

Late 1990s–2000s: host-based monitoring platforms

Example systems:

  • Nagios (1999)
  • Cacti (for web graphing)

Typical model:

  • Maintain a list of hosts, regularly poll, and alert when something breaks.

“Use” methodology (resource-centric, Brandon Gregg)

For each resource (CPU/memory/disk/network), monitor:

  1. Utilization (how busy)
  2. Saturation (queued/buildup beyond capacity)
  3. Errors (error count)

This matches the classic “host/resources are fixed and enumerable” world.

3) What exactly gets monitored (layer model)

  • Hardware/infrastructure: CPU, memory, disk (including speed/latency), temperature, power, network.
  • OS/host layer: load average, running processes, file descriptors, swap, live services.
  • Network layer: availability, packet loss, latency, port status (SNMP legacy).
  • Application performance monitoring (APM): response time, throughput, error rate, request tracing through services.
  • Databases: queries per second, slow queries, connection pool size, replication, caching-related metrics (e.g., Varnish mentioned).
  • Containers/Kubernetes
  • Business metrics (top layer): signups/orders/revenue/conversions.
    • Key point: a system can look “green” technically yet fail business outcomes (e.g., orders = 0 while dashboards show no obvious system errors). Good monitoring includes money.

4) User perspective approaches

  • Synthetic monitoring: automated “robots” simulate user journeys from multiple world locations.
  • RUM (Real User Monitoring): collects telemetry directly from real browser users.

Synthetic monitoring finds problems earlier; RUM shows what real users experience.

5) Why old monitoring struggled with containers/microservices (2010s shift)

  • Containers are ephemeral: dynamic addresses, frequent restarts, changing “host lists.”
  • Pre-registering targets (host-based polling paradigm) becomes ineffective.
  • Solution presented: Prometheus (originated at SoundCloud in 2012; later publicized in 2015).

Prometheus innovations

  • Pull/scrape model (“pool model”): Prometheus discovers targets and scrapes metrics via an HTTP endpoint (referenced as /metrics).
    • If a target doesn’t respond, unavailability is known immediately—no separate check required.
  • Multidimensional labels: metrics are time series with (key, value) label pairs (method, status code, service name, etc.).
  • TSDB (time-series database) designed for metrics storage.
  • Ecosystem: Grafana as the de facto dashboard layer (metrics queried from Prometheus).

“RED” methodology (service/user-centric, Tom Wilkie)

For each service, monitor:

  1. Rate (requests per second)
  2. Errors
  3. Duration (request latency)

Framed as complementing “USE”: USE diagnoses which resource is broken; RED reflects whether user-facing outcomes are bad.

6) From monitoring to observability (three pillars + standard)

  • Monitoring: answers predefined questions (works if you know in advance what will break).
  • Observability: lets you answer questions you didn’t know to ask beforehand.

Pillars:

  1. Metrics: what is happening (fast, chart/alert friendly, but limited detail)
  2. Logs: what exactly happened at the moment (diagnostic detail)
  3. Traces: where requests fail/slow across multiple microservices (end-to-end path with timings)

Standard mentioned:

  • OpenTelemetry (merged lineage from OpenCensus and OpenTracing; vendor-neutral way to send metrics/logs/traces).

Continuous profiling (new/4th pillar)

  • Uses Linux eBPF to capture call stacks without code changes or restarts.
  • Answers “why” (which code line causes slowdown).
  • Example described: intermittent slowdowns traced to a third-party process inside a container added by a cloud provider—something metrics/logs/traces couldn’t reveal.

7) Practical limitation: cardinality explosion (Prometheus labels)

  • Prometheus stores a separate time series for every unique label combination.
  • Adding high-cardinality labels like user ID, request ID, full URL parameters can create massive series and overwhelm memory (cardinality explosion).
  • Guidance:
    • Use labels with few possible values (method, code, service name, region).
    • Rely on logs/traces for unique identifiers.

8) Turning signals into reliability: SLI/SLO/SLA + error budget

Concepts introduced (often interview topics):

  • SLI: measurable health indicator (e.g., successful requests proportion)
  • SLO: internal target (e.g., 99.9% successful in 30 days)
  • SLA: customer-facing contract; described as softer than SLO

Error budget

  • If your SLO is 99.9%, you can “spend” the remaining 0.1% on experiments.
  • If the budget is consumed too quickly, teams switch toward reliability work until restored.

9) Alerting culture: avoid alert fatigue

  • The problem isn’t too few alerts—it’s too many, causing engineers to ignore them (alert fatigue).

Rules emphasized:

  1. Every alert must have clear required actions; otherwise it’s just a dashboard metric.
  2. Limit incidents per on-call duty shift (target stated: no more than ~2 incidents per shift).

Burn Rate alerting

  • Alert based on how quickly error budget is being consumed.
  • Burn rate = 1 → on pace; much faster → immediate action.

After incidents:

  • Use postmortems to identify systemic causes and prevent recurrence.

10) Tooling landscape (mentioned stacks)

Metrics/monitoring

  • Prometheus
  • VictoriaMetrics (compatible, lower memory usage for huge volumes)
  • Long-term storage: Mimir or Thanos
  • Dashboards: Grafana

Logs

  • ELK stack: Elasticsearch + Logstash + Kibana
  • Mentioned open fork: OpenSearch
  • Grafana Loki approach (“Prometheus for logs”):
    • Indexes labels only, not full text → cheaper

Tracing

  • Jaeger
  • Grafana Tempo

All-in-one platforms (paid)

  • DataDog and other examples; pricing can include hosts, data volume, and custom metrics.

Legacy/other familiar tools

  • Zabbix
  • Nagios

Main speakers/sources

  • The narration is from the Simply DevOps channel (speaker not individually named).
  • Technical attributions mentioned:
    • Brandon Gregg (USE methodology)
    • Tom Wilkie (RED methodology)
    • OpenTelemetry lineage: Google OpenCensus and OpenTracing (merged into OpenTelemetry)
  • Tool/company references:
    • SoundCloud (Prometheus origin)
    • CNCF (Prometheus noted as second project after Kubernetes)

Original video