Optimising Backend Apis | Using Redis Queue for Cron Jobs Spring | Backend Challeng Series

High-level purpose

Practical walkthrough from a “Backend Challenge” series showing how to handle long-running/background tasks (example: bulk email sending) using a queue-based architecture with Redis and Spring Boot. Emphasis on moving heavy work out of the request/response flow so APIs remain non-blocking and scalable.

Architecture and components

Flow: Client -> Spring Boot API controller -> Email producer -> Redis list (queue) -> Email worker (consumer) -> actual email send

Main components:

• Controller: receives bulk email requests and delegates to the service.
• Producer: pushes each email request into a Redis list (queue) using RPUSH.
• Redis queue: a list (single key) storing EmailRequest objects.
• Consumer (worker): scheduled background task that polls Redis (LPOP) and calls sendEmail() for each item.
• Email sender: mocked in the demo (sleep to simulate work). For real usage, use Spring Boot mail / JavaMail.

Dependencies and libraries:

• spring-boot-starter-data-redis
• Redis client (e.g., Jedis)
• RedisTemplate (or a typed template)
• For real email delivery: spring-boot-starter-mail / JavaMail

Configuration notes:

• Redis configuration was referenced from a separate caching video (not re-implemented in the demo).
• Scheduling must be enabled with @EnableScheduling in the main/config class.

Implementation details and tips

Controller

• Exposes a REST endpoint (e.g., POST /api/v1/emails) that accepts a list of EmailRequest DTOs via @RequestBody.
• Delegates the list to a bulk email service.

Producer/service

• pushEmailsToQueue loops through the provided list and pushes each EmailRequest into Redis under a queue key (e.g., EMAIL_Q).
• Example operation: redisTemplate.opsForList().rightPush(queueKey, item).

Consumer/worker

• Implemented as a scheduled method annotated with @Scheduled (e.g., fixedDelay every 2–5s).
• Polls Redis using redisTemplate.opsForList().leftPop(queueKey) and processes the popped item if non-null.
• sendEmail() is called for each consumed EmailRequest; in the demo this is mocked (sleep), but in production it should use a mail sender.

Error handling and retries

• Demo shows basic try/catch around send logic.
• Recommended to implement retry mechanisms if sending fails (either in consumer logic or inside sendEmail).

Debugging note

• A ClassCastException occurred in the demo due to importing the wrong EmailRequest class; fix by importing the DTO from the correct package.

Scheduling options

• fixedDelay, fixedRate, or cron expressions — choose based on desired semantics.

Benefits highlighted

• Prevents blocking the API: clients receive immediate acknowledgment that the campaign started.
• Scalability: decouples producers and consumers so large volumes (thousands) can be handled.
• Independent scaling: workers can be scaled separately from the API.
• Easier failure handling: retries and failure strategies are simpler to implement in background processors.

Limitations & suggested improvements

Limitations of the simple Redis-list + scheduled polling approach:

• Polling wastes CPU cycles when the queue is empty because the scheduled task keeps running.
• Delivery semantics, persistence, and advanced retry/ack handling are limited.

Suggested production-grade improvements:

• Use a proper message broker with event-driven listeners, e.g. RabbitMQ or Kafka (or Redis Streams).
  • Benefits: consumer groups, persistence, better delivery semantics, native listeners (no constant polling), more robust retry/ack patterns.
• Use Redis Streams or pub/sub models (better than list + scheduled polling).
• Consider separating producer and consumer into distinct services for clearer boundaries and independent deployment.

Tutorials / related resources mentioned

• This video is part of a 50-backend-concepts series (practical + theoretical videos).
• A prior caching video: “Make API 100x faster vs 10 Cache Hit vs Cache Miss, step by step implementation” — contains the Redis configuration details referenced.
• An upcoming/paid course (covers microservices, DevOps, AWS and a real-world project) — link promised in the video description.

Key code/configuration pointers

• Redis template: RedisTemplate<String, Object> (or a typed template).
• Queue key: store items under a single queue key (e.g., EMAIL_Q).
• Producer push: redisTemplate.opsForList().rightPush(queueKey, item).
• Consumer pop: redisTemplate.opsForList().leftPop(queueKey) inside an @Scheduled method.
• Enable scheduling: add @EnableScheduling in your config or main class.
• For real email sending: add spring-boot-starter-mail and configure SMTP/email provider properties.

Main speaker / source

• The summary comes from the video’s author/presenter (host of the Backend Challenge / YouTube series). The presenter references other videos (caching) and a paid course. Technologies used: Spring Boot (Spring Data Redis, Scheduling), Redis (Jedis client), JavaMail / spring-boot-starter-mail (optional).