Object, Encodings, and Implementing INCR | Redis Internals

Key technological concepts & observations

Redis INCR vs GET return types

• INCR myKey
  • Demo behavior: INCR myKey creates the key if missing and returns an integer (e.g., 1).
• GET myKey
  • Then GET myKey returns a string representation (e.g., "11").

Explanation: Redis doesn’t have a separate “int type” at the object level. Instead, integers are stored as strings with an appropriate encoding. Internally, the process effectively does:

• string → integer → increment → integer → string

Standard Redis object design

Redis stores values as a Redis Object that wraps metadata around the underlying data.

• Each value lives in Redis’s main dictionary/hash table:
  • Key is hashed to a slot
  • Value is a Redis object
• Redis Object fields (as described):
  • type
  • encoding
  • LRU info for eviction
  • reference count (reference counting / garbage collection)
  • pointer to the concrete underlying structure (e.g., string/list/hash representations, bloom filter, etc.)

Size/memory emphasis

• type and encoding are stored using bit fields (4 bits each) to save memory.
• Overall is described as about ~12 bytes per Redis object, including LRU, refcount, and pointer components.
• Main takeaway: memory efficiency via compact metadata layout.

Types vs encodings

Supported types (per subtitles)

• string, list, set, sorted set (zset), hash, module (mentioned as not a “core type” in the same sense), and “string” again (likely a subtitle duplication).

No “int type”

• There is no separate “int type”.
• “Integer-ness” is represented by string type + integer encoding.

Encoding represents the concrete implementation

Examples mentioned:

• raw (bytes)
• string (string representation)
• int (stored as string bytes but tagged with integer encoding)
• embedded string for small strings (sub-44 bytes threshold described)
• List: zip list vs linked list
• Hash: zip list vs hashtable
• Sorted set: ziplist vs skiplist

Automatic representation switching

• Smaller structures use compact encodings (ziplist / embedded string).
• When size thresholds are exceeded, encodings change to more scalable representations (e.g., ziplist → linked list).

---

Tutorial / guide-style implementation described (re-implementing Redis internals)

Reference implementation source

• The speaker points to a GitHub source: github.com/tidb/dice
• They perform an “exhaustive code walk-through” of implementing types and encodings, specifically to support INCR.

Step 1: Create/modify object.go

• Expand object metadata beyond just value and expiresAt.
• Add:
  • type
  • encoding
• Use an 8-bit field (uint8) where:
  • upper 4 bits = type
  • lower 4 bits = encoding
• Motivation: Go doesn’t have C-style bit fields, so they emulate the packed layout.

Step 2: Add typeencoding.go helpers

• Functions to:
  • extract type from the high 4 bits
  • extract encoding from the low 4 bits
  • validate constraints:
    • assert type
    • assert encoding
• Highlighted edge-case behavior:
  • INCR must only work when:
    • the object’s type is string
    • the object’s encoding corresponds to integer

Step 3: Modify store

• When creating a new stored object, compute and set:
  • objectType (likely always string in the simplified model)
  • objectEncoding (based on how the incoming value should be interpreted)

Step 4: Implement “deduce type encoding” for SET

• On SET, the system attempts to parse the value as an integer:
  • If convertible to int:
    • store encoding = integer
    • still store the underlying data as string bytes
  • If not convertible:
    • choose embedded string if length < 44
    • otherwise choose raw
• Key idea: integer encoding is metadata, not a separate integer object type.

Step 5: Implement INCR in eval

Behavior implemented to match Redis:

• If key missing:
  • create it with:
    • value = 0
    • no expiry (as described)
    • type = string
    • encoding = integer
• If key exists:
  • verify type is string
  • verify encoding is integer
  • parse value string → integer
  • increment
  • convert back to string
  • update stored value
  • return increment result as an integer (matching “INCR returns integer, GET returns string”)

---

Demonstrated outcome

• Testing the re-implementation shows:
  • INCR K returns numeric increments (1, 2, 3, …).
  • GET K returns values in string form but with metadata indicating encoding is integer.
  • On every increment, the logic checks encoding constraints before converting and incrementing.

---

Design/analysis takeaways (why Redis does this)

Extensibility via types + encodings

Because Redis stores values mostly as bytes with metadata, you can implement new representations as new encodings.

• Example idea: represent a Bloom filter as:
  • type = string
  • encoding = bloom filter
  • underlying storage handled via the pointer to that representation

Tradeoff acknowledged

• Conversions add cost (string ↔ int), but Redis chooses this for:
  • simplicity/user experience
  • uniformity of value handling
• Alternative mentioned: inventing a true integer type, but Redis avoids that intentionally.

---

Main speakers / sources

• Speaker: Unnamed (single presenter explaining Redis internals and Go implementation)
• Source referenced: Redis source code concepts (types/encodings/object layout) and the implementation scaffold at github.com/tidb/dice