Godot Source Code explained by Technical Lead 02: Memory Management

Main ideas & lessons (memory management in Godot / general engine practice)

1) Why memory allocation strategy matters

• Core engine systems make many decisions based on how memory is allocated and managed.
• To understand engine behavior and performance, you need the underlying theory of memory management.

2) How typical heap allocation works (concepts)

• Computers have limited physical memory (bytes arranged conceptually like a grid).
• Program allocations are “random” in the sense that allocation order/size patterns are not predictable.
• In C/C++-style allocation (e.g., malloc/free, new/delete):
  • Allocations usually occur on the heap.
  • Allocation returns a pointer.
  • The allocated pointer must be used later to free the memory; you can’t “move” the pointer.

3) Heap fragmentation and why it can cause out-of-memory

• The heap generally grows from some starting point.
• With mixed-size allocations:
  • Frequent allocations and frees create holes in the heap.
  • New allocations may reuse holes instead of always appending at the end.
• Fragmentation:
  • You can have “enough total free memory” but still fail to allocate a contiguous block large enough.
• Result:
  • On systems without sufficient contiguous space (or limited memory without swap), the program can hit out-of-memory and crash.

4) Common mitigation strategies (traditional approach)

For small allocations

• Engines often reserve a portion of the heap so that small allocations don’t push the system into fragmentation failure.
• Typical idea:
  • Keep a percentage of heap free.
  • If the game doesn’t exceed a threshold usage, you avoid “too fragmented to allocate” failures.

For large allocations (pool/compaction strategy)

• A strategy described for large allocations is pool memory:
  • Reserve a region for large blocks (a pool).
  • Allocate within the pool.
  • Use locking/handles so compaction is safe:
    • If the pool must be compacted due to fragmented space, blocks can be:
      • compacted if not in use
      • not moved (or delaying compaction) if currently locked/in use
• Claimed outcome (Godot 4 described as “how Godot 3 works”):
  • Pool compaction can resolve fragmentation for large allocations while keeping safety via handles/locking.

5) Why fragmentation is less of a problem on modern 64-bit systems

• On modern CPUs (64-bit):
  • There is a virtual address space that can be vastly larger than physical RAM.
  • Physical memory pages can be mapped into many places in virtual space.
• Memory allocations can be placed in regions that reduce practical fragmentation issues.
• Large allocations typically map using pages, and pages can be mapped flexibly.
• Takeaway:
  • The speaker argues that with 64-bit virtual memory, fragmentation is “not really a problem” in modern OSes, including for Godot 4.

6) Performance is still the real challenge

Even if fragmentation isn’t a major issue anymore, performance pitfalls remain:

• Slow allocations:
  • “Malo/new/free” style general-purpose allocation may search for holes and become very slow.
• Sparse access / poor locality:
  • If memory is scattered across address space, access becomes cache-inefficient and slow.

7) CPU cache/memory hierarchy: what matters for engine performance

Cache levels and typical latency trends (conceptual)

• Global RAM: slowest access.
• L3 cache (per CPU package): large (tens to hundreds of MB), much faster than RAM.
• L2 cache (per core): smaller (often ~hundreds of KB), faster than L3.
• L1 cache (per core): tiny and fastest (around ~1 ns scale, very small).

Cache lines (critical concept)

• When reading from RAM/caches, CPUs fetch in cache lines, often 64 bytes.
• Implication:
  • If your data access is sparse/random, each “small” access still pulls a full cache line.
  • Organizing data to improve locality reduces wasted cache-line transfers.

Pages & address translation (TLB)

• The virtual address space is mapped to physical memory through pages.
• The CPU uses a TLB (Translation Lookaside Buffer) to translate virtual pages to physical pages efficiently.
• If memory is scattered and not page-aligned/grouped well:
  • more translation work is needed
  • performance suffers.
• Takeaway:
  • Grouping allocations so they fit typical page sizes (e.g., ~4 KB) can improve performance.

8) How to optimize for each cache level (instruction-like guidance)

L1 cache optimization guidance

• Treat L1 as very tiny (few KB).
• Use simple, sequential access patterns:
  • “Pac-Man style”: do one thing after another on contiguous/sequential data.
• Avoid operations that likely break locality:
  • complex logic scattered around
  • calling functions that may evict/reload L1 working data
• When not strictly sequential:
  • align/structure data around 64-byte cache lines.

L2 cache optimization guidance

• L2 is a larger per-core working area (speaker suggests ~128 KB typical scale).
• Suitable for more complex algorithms than L1:
  • hashing, decoding, decompression, VM/interpreters, etc.
• Guidance:
  • Keep active working sets near or under L2 size per batch.
  • It’s fine to have conditionals as long as the working set remains cache-resident.

L3 cache optimization guidance

• L3 is shared across cores in the CPU package (global per CPU on that chip).
• For frame-based game engines:
  • Aim to keep “what you touch every frame” inside L3 as much as possible.
• Guidance:
  • keep data packed and grouped
  • avoid lots of allocations scattered across the engine
  • reduce car allocations of “unrelated” objects; use pooling/packing approaches.

9) Why “ECS = everything sequential” is overstated

• The speaker claims only a small fraction (~5%) of typical game behavior benefits from extreme L1-level optimization.
• In most cases:
  • focusing on engine-wide packing (especially L2 and L3 efficiency) is more generally valuable than micro-optimizing for L1 everywhere.

---

Methodologies / strategies specifically recommended (detailed bullets)

A) Avoid fragmentation and out-of-memory (traditional heap approach)

• Small allocations
  • Leave a reserved/free percentage of heap unused to reduce the likelihood of fragmentation preventing large contiguous allocations.
• Large allocations
  • Use a pool for large blocks.
  • Allocate large blocks inside the pool.
  • Employ locking/handles so:
    • compaction can occur when needed
    • but blocks in use are not moved until unlocked.

B) Reduce allocation and improve performance locality (modern engine practice)

• Prefer custom allocators over generic malloc/new/free because:
  • generic ones can be inefficient (hole search)
  • generic allocations can lead to sparse memory placement.

C) Cache-friendly data layout approach

• Always structure/arrange data access around:
  • 64-byte cache lines
  • avoid scattered access on the heap
• L1:
  • keep working data tiny
  • use sequential reads/writes
  • avoid complex branching that thrashes L1
• L2:
  • keep working set around typical L2 size (speaker suggests ~128 KB)
  • acceptable to have conditionals/complex logic if working set fits
• L3:
  • pack “per-frame accessed data” so it fits in L3 (megabytes range)
  • consolidate/pool data to reduce random global heap accesses.

D) Godot-style allocator strategies (as described)

1) Growth-only allocation (worst-case allocator concept)

• Use containers that only grow:
  • capacity increases on growth
  • removing elements typically does not shrink capacity
• Benefits:
  • very few heap allocations (capacity growth causes allocations; steady-state reuses already reserved memory)
  • memory becomes contiguous for each container
  • fewer fragmentation issues because related allocations often happen together (same sizes / same “batch lifetime”)
• Downsides (acknowledged):
  • memory usage tracks worst-case peak; unloading may not reclaim capacity
  • peak memory can increase if different parts of the game peak at different times.

2) Paged allocators (page-based growth-only)

• Allocate memory in pages (chunks):
  • expanding needs adds a new page rather than relocating an entire buffer
• Benefits:
  • avoids expensive “resize and move all data” behavior
  • aligns well with cache/pages
• Bookkeeping:
  • keep a page list and a free list of available elements/slots.
• Allocation/free cost:
  • allocation and freeing within existing pages is extremely cheap
  • creating a new page may involve heavier synchronization (e.g., mutex), but is infrequent.

3) Cache-friendly hash tables: open addressing (growth-only)

• Use open addressing for hash tables:
  • data clustering is expected; to maintain lookup performance:
    • keep occupancy below a threshold (speaker suggests keeping ~25% free if using ~75% capacity)
• Requirements:
  • good hash functions with uniform distribution are crucial.
• Change described:
  • replace poor-distribution hash (speaker mentions djb2 as a bad idea)
  • move to m3 (better distribution) to reduce clustering bottlenecks.
• Lifecycle constraint:
  • hash tables are growth-only while in use
  • shrinking typically isn’t automatic; reset/free requires manual handling.

---

Speakers / sources featured

• Speaker: “Technical Lead” (Godot Technical Lead), implied by the video title (“Godot Source Code explained by Technical Lead 02”).
• No other named individuals or external sources are explicitly credited in the subtitles (only general references like operating systems, CPUs, and concepts such as TLB, cache lines, etc.).