Video summary

Implementing AOF Persistence | Redis Internals

Main summary

Key takeaways

Technology

Redis Persistence Overview (AOF)

Redis is often misunderstood as “in-memory only,” but it supports persistence.

Redis persistence comes in two formats:

1) RDB (Redis Database)

  • Created as a point-in-time snapshot of the dataset.
  • Triggered by configuring a flush/snapshot frequency (e.g., every 5 minutes).
  • Redis uses forking during RDB generation:
    • The main server continues serving requests.
    • A child process dumps the in-memory structures to disk.
  • Tradeoff: RDB rewrites full snapshots.
    • Frequent snapshots are costly.
    • If the server crashes between snapshots, you lose data since the last snapshot window (e.g., up to ~5 minutes).
  • Output is compact and typically a single file you can copy/move (even to locations like S3/Drive).

2) AOF (Append-Only File)

  • Acts like a commit log / binlog:
    • Logs write operations (not reads like GET).
  • The AOF is append-only and stores commands in RESP-encoded (command-like, human-readable) form rather than a binary format.
  • Durability tradeoff:
    • With frequent AOF flushing (example: once per second), data loss is reduced to about the last flush interval (e.g., ~1 second).

AOF Specifics

Growth and compaction

  • AOF can grow very large if every write is logged indefinitely.
  • Mitigation: periodic AOF rewrite (log compaction) using a background mechanism like BGREWRITEAOF.
    • Redis rewrites the AOF in a more efficient representation.
    • Example:
      • Repeated SET k V1 -> V2 -> V3 -> V4
      • After rewrite, the AOF may only keep the latest effective state (e.g., SET k V4), shrinking the file.

Crash recovery

  • Because the AOF records writes, Redis can reconstruct the dataset on startup by replaying the AOF.

Data integrity verification

  • Uses redis-check-aof to verify whether an AOF file is valid and repair it if needed.

Background Rewrite Safety / Atomicity

  • During background rewrite, Redis writes the new log to a temporary file first.
  • After completion, it renames the temp file to the configured AOF filename (commonly appendonly.aof), avoiding disruption to ongoing writes.

Implementation Walkthrough (Conceptual Code Changes)

The walkthrough then demonstrates implementing AOF rewrite in a Redis-like codebase.

Added command

  • Introduces a command: BG rewrite aof
  • Implemented via an eval function (for the demo).

Demo approach

  • The described implementation is synchronous in that specific step (i.e., it doesn’t truly fork a background process there), focusing first on core functionality.

Main components

  • A new AOF module/file (mentioned as aof.go).
  • A function like dump all aof that:
    • Opens/creates an output file (named via config, e.g., something like dice-master.aof).
    • Iterates through all current in-memory key/value pairs.
    • Writes them into the AOF using the same RESP command encoding format clients would use.

RESP Encoding Behavior for AOF Entries

  • The AOF rewrite outputs commands equivalent to what the CLI would issue.
  • Example:
    • For key k and value v, the rewrite outputs:
      • SET k v
  • Commands are encoded as RESP arrays in the format:
    • *<numargs>\r\n$<lenarg1>\r\n<arg1>...\r\n

Result:

  • The rewritten AOF becomes a replayable command stream, compacted to represent the current dataset state.

Live Demonstration / Verification Steps

  1. Start the server.
  2. Clear any prior AOF file.
  3. Set multiple keys.
  4. Trigger BG rewrite aof.
  5. Inspect the generated AOF file:
    • It contains RESP-encoded SET commands for the current keys.
    • If a key was set multiple times before rewrite, the rewritten AOF includes only the latest value, since it reflects the in-memory dataset at rewrite time.
  6. Validation:
    • Run redis-check-aof on the generated file to confirm it is valid.
  7. Use the AOF on restart to reconstruct state by replaying it.

Pros / Cons Summary

  • Advantage (AOF):
    • Higher durability and lower potential data loss vs RDB
    • Example given: ~1 second vs ~5 minutes
  • Disadvantage (AOF):
    • AOF files are typically larger than RDB files because they store more detail
    • Growth must be managed via rewrite/compaction
  • The implementation is presented as Redis compliant, emphasizing use of Redis tooling to test correctness.

Speakers / Sources

  • Speaker: Not explicitly named in the subtitles (single instructor narrator).
  • Referenced tools/commands: Redis persistence concepts and utilities, especially redis-check-aof, and the concept of BGREWRITEAOF.

Original video