Kafka Fundamentals: Understanding Segments, Commit Log, and Retention Policy

segments in Kafka are sets of messages stored within partitions.

messages are appended to segments as new messages come in.

The commit log in Kafka stores actual data produced by producers.

The retention policy in Kafka determines how long data should be kept before deletion.

There are two types of retention policies: size-based and time-based.

The cleaner process runs in the background to delete old data based on the retention policy.

Default retention period in Kafka is 168 hours or 7 days.

segments are created as new messages come in, and old segments are deleted to maintain partition size limits.