Apache Kafka 101: ksqlDB (2023)

Main Speaker

• Tim Berglund, Confluent

The video features Tim Berglund from Confluent discussing ksqlDB, a specialized database for stream-processing applications built on Apache Kafka. Here are the key technological concepts and features highlighted:

• Integration with Kafka: ksqlDB operates outside the Kafka cluster, running on its own servers that interface with Kafka brokers. This allows for scalable and fault-tolerant stream-processing jobs.
• SQL-Based Stream Processing: Users can write stream-processing jobs using SQL, making it accessible for those who may not have a Java background. The jobs can be submitted and queried via a REST interface.
• Ease of Use: ksqlDB can be run locally using a standard Docker image, making it easy to develop and test applications without needing extensive infrastructure.
• Concise Code: Compared to Kafka Streams, ksqlDB allows for more readable and compact code for similar operations, which can be beneficial for developers familiar with SQL.
• Data Integration: ksqlDB integrates with Kafka Connect, enabling users to connect to external data sources and configure connectors directly within the ksqlDB environment.
• Continuous Processing: It functions as a continuous processing engine for event streams, providing database-like access to the results of stream processing.
• Not a Replacement for Traditional Databases: While ksqlDB offers database-like functionalities for event streams, it is not intended to replace traditional databases like PostgreSQL.

Key Takeaways

• ksqlDB is optimized for stream-processing with a SQL interface.
• It simplifies the development process for applications interacting with Kafka.
• It integrates seamlessly with existing Kafka and Kafka Connect setups.