Kafka Error Handling with Spring Boot | Retry Strategies & Dead Letter Topics | JavaTechie

Summary (Kafka Error Handling with Spring Boot: Retry Strategies & Dead Letter Topics)

The video explains how to handle errors in a distributed Kafka setup (multiple containers/machines), where the producer → Kafka → consumer processing pipeline can fail due to temporary infrastructure issues (e.g., database connections down). The goal is to avoid losing events and to provide a strategy to recover, retain failed messages, and support later investigation or reprocessing.

Core Concepts / Features Covered

1) Retry failed Kafka events

• When a consumer fails to process a message (e.g., validation throws an exception), Kafka—via Spring’s retryable-topic support—should automatically retry processing.
• Example behavior: configuring retries = 4 results in multiple sequential attempts (retry attempts are performed accordingly across tries).
• If processing succeeds during any retry attempt, the message is eventually handled successfully.

2) Dead Letter Topic (DLT) for exhausted retries

• If the message still cannot be processed after the configured retry count is exceeded, it is sent to a Dead Letter Topic (DLT).
• A DLT is a separate topic used to store failed events/messages so they can be:
  • monitored,
  • investigated,
  • and potentially reprocessed later.

Tutorial / Demo Implementation (Spring Boot)

Example use case

• Processing Finance transaction-like events.
• A User object is published to Kafka and consumed.
• The consumer includes validation logic that intentionally throws an exception when a user’s IP address is in a restricted list (to simulate failure).

Components demonstrated

• Publisher: takes a User object and publishes it to a Kafka topic.
• Consumer: listens to the topic and validates the input; throws exceptions to trigger retry logic.

Kafka configuration / topics

• The tutorial includes creation of a main topic with multiple partitions (shown as 3 partitions).
• Uses console/logs and a Kafka visualization tool (Kafka Explorer) to verify where messages land.

Spring Kafka Annotations / Flow

@RetryableTopic (retriable topic mechanism)

• Add the annotation to enable retry behavior.
• Setting retry attempts causes Kafka/Spring to create internal retry topics (with suffixes like retry-0, retry-1, retry-2, etc.).
• Each retry topic is used for the next attempt.

@DltHandler (DLT handling)

• A method annotated with @DltHandler captures messages that still fail after retries are exhausted.
• The handler logs details and confirms the message reached the DLT.

Validation / Testing Scenarios Shown

Happy path

• Send a valid IP:
  • consumer processes normally
  • no DLT record is produced.

Failure path with retries

• Send a restricted IP:
  • first consumption fails,
  • message is retried across multiple retry topics,
  • and finally ends up in the DLT after retries are exhausted.
• The tutorial verifies retries by checking logs that include:
  • topic name for each attempt (including retry topics),
  • offset information.

Bulk publish scenario (CSB/JSON file → 100 users)

• Loads ~100 user records from a file and publishes them one-by-one.
• Only 4 users have restricted IPs, so:
  • only those 4 should end up in the DLT,
  • confirmed by DLT topic message counts and Kafka Explorer output.
• Emphasis: this approach avoids manually creating failure topics—Kafka + Spring manage DLT and retry topics automatically using annotations.

Extra Configuration Options Mentioned

• Backoff strategy for retry timing:
  • delay (e.g., 3000 ms),
  • multiplier (e.g., 1.5),
  • max delay (e.g., 15000 ms).
• Ability to exclude retries for specific exception types (e.g., not retry on certain runtime exceptions like NullPointerException).

Main Speaker / Source

• JavaTechie (the creator guiding through the demo; intro/outro mention “Java tii” / “JavaTechie”).