Apache Kafka: Understanding Key Concepts and Components.

In the world of modern distributed systems and data processing, Apache Kafka (commonly known as Kafka) has emerged as a powerful and widely adopted platform for real-time data streaming and event processing. Specifically, Apache Kafka is extensively used in microservices architectures.

In this article, we will delve into the components of Kafka, explore how Kafka works, and understand its advantages and disadvantages, as well as its applications in real life.

What is Apache Kafka?

Apache Kafka is an open-source stream-processing software platform developed by LinkedIn and donated to the Apache Software Foundation, written in Scala and Java.

“Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.”

Source: Kafka home page

The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Its core architectural concept is an immutable log of messages that can be organized into topics for consumption by multiple users or applications. A file system or database commit log keeps a permanent record of all messages so Kafka can replay them to maintain a consistent system state.

Kafka stores data durably, in serialized fashion. It distributes data across a cluster of nodes, providing performance at scale that’s resilient to failures.

Kafka is available as open source code or from vendors who offer it as a cloud managed service, including AWS (MKS – Managed Kafka Service), Azure, Google and Confluent (Confluent Cloud), the company founded by the creators of Kafka.

What is Apache Kafka Used For?

Kafka is flexible and widely-used, but there are several use cases for which it stands out:

• Log/Metricsk, aggregation: Kafka provides log or event data as a stream of messages. It removes any dependency on file details by gathering physical log files from servers and storing them in a central location. Kafka also supports multiple data sources and distributed data consumption.

• Stream processing: Kafka is valuable in scenarios in which real-time data is collected and processed. This includes raw data that’s consumed from Kafka topics and then enriched or processed into new Kafka topics for further consumption as part of a multi-step pipeline.

• Commit logs: Any large-scale distributed system can use Kafka to represent external commit logs. Replicated logs across a Kafka cluster help data recovery when nodes fail.

• Activity tracking: Data from user click stream activities, such as page views, searches, and so on, are published to central topics, with one topic per activity type.

• Event Sourcing: Event-driven architectures benefit from Kafka's ability to capture and store events that represent state changes, enabling the implementation of reliable and scalable event sourcing.

• Data Integration: Kafka acts as a bridge between different systems, enabling seamless data integration between applications and databases.

Understanding Event-driven Architecture in Apache Kafka

Kafka receives and sends data as events called messages which are organized into topics which are “published” by data producers and “subscribed” to by data consumers, which is why Kafka is sometimes called a “pub/sub” system. An event is simply a statement that something occurred in the real world. An event may also be referred to as a record or a message.

What is a Kafka cluster?

A cluster is a distributed computing concept in which a group of devices works together to achieve a specific purpose. Multiple Kafka brokers form a Kafka cluster.

The main goal of a Kafka cluster is to spread workloads evenly across replicas and partitions. Kafka clusters can scale without interruption. They also manage the persistence and replication of data messages. If one broker fails, other Kafka brokers step in to offer similar services without data loss or degraded latency.

Kafka Cluster Architecture

Multiple components comprise the core Kafka cluster architecture:

• Kafka Events

• Broker

• Controller

• Partitions

• Consumer

• Producer

• Topic

• Zookeeper (phasing out)

• Schema registry

About the Apache Kafka Events

A Kafka event records the fact that "something has happened" in the world or within your business. It is also known as a record or message. In many Kafka documents, the terms event, record, and message are used interchangeably.

Reading and writing data in Kafka are done through events. Each event contains a key, value, and metadata (if any).

Example of an event containing a key, value, and metadata (timestamp):

{
  "key": "user123",
  "value": {
    "action": "login",
    "status": "success"
  },
  "metadata": {
    "timestamp": "2024-07-20T10:15:30Z"
  }
}

About the Apache Kafka broker

A broker is a single Kafka server. Kafka brokers receive messages from producers, assign them offsets, and commit the messages to disk storage. An offset is a unique integer value that Kafka increments and adds to each message as it’s generated. Offsets are critical for maintaining data consistency in the event of a failure or outage, as consumers use offsets to return to the last-consumed message after a failure. Brokers respond to partition call requests from consumers and return messages that have been committed to disk. A single broker operates based on the specific hardware and its functional properties.

About the Apache Controller

Kafka brokers form a cluster by directly or indirectly sharing information. In a Kafka cluster, one broker serves as the Apache controller. The controller is responsible for managing the states of partitions and replicas and for performing administrative tasks such as reassigning partitions and registering handlers to be notified about changes.

Although the Controller service runs on every broker in a Kafka cluster, only one broker can be active (elected) at any point in time. The Kafka Controller is created and starts up as soon as the Kafka server starts up.

About partitions in Apache Kafka

Partitioning is a foundational principle in most distributed systems. In Kafka, a topic is split into multiple partitions. A partition is a discrete log file. Kafka writes records to each partition in append-only fashion. In other words, all of the records belonging to a particular topic are divided and stored in partitions.

Kafka distributes the partitions of a particular topic across multiple brokers. This distributed layout improves scalability through parallelism, as you’re not limited to one specific broker’s I/O. Kafka also replicates partitions and spreads the replicas across the cluster. This provides robust fault tolerance; if one broker fails other brokers can take over and pick up where the failed machine left off.

When a new message is written to a topic, Kafka adds it to one of the topic’s partitions. Messages with the same key (for example, an enrollment number or customer ID) are published to the same partition. Kafka assures that any reader of a given topic/partition always consumes messages in their published sequence.

Each event in a partition is assigned a unique offset, starting from 0 for the first event in the partition and incrementing by one for each subsequent event. The offset is used to determine the position of the event within a partition.

Each partition has a broker designated as the leader, which owns that partition, while the remaining brokers store replicas of the partition, known as followers.

If the leader broker fails, one of the followers with updated data will be selected as the new leader. This process is called leader failover, and it ensures the availability of the data.

About consumers in Apache Kafka

Consumers are applications or machines that subscribe to topics and process published message feeds. Sometimes called “subscribers” or “readers” consumers read the messages in the order in which they were generated.

A consumer uses the offset to track which messages it has already consumed. A consumer stores the offset of the last consumed message for each partition so that it can stop and restart without losing its place.

Consumers interact with a topic as a group (although a group can consist of only one consumer). This enables scalable processing. The group ensures that one member only consumes each partition. if a single consumer fails, the group’s remaining members reorganize the consumed partitions to compensate for the absent member. Groups enable Kafka to consume topics with a massive amount of messages horizontally.

Consumers connect to Kafka Brokers via the TCP network protocol. This is a bi-directional connection.

When a consumer group reads events from partitions, there are three possible scenarios:

• Scenario 1: The consumer group has fewer consumers than the number of partitions in a topic.

Consumers 1 and 2 read events from 4 partitions of Topic T1 sequentially.

• Scenario 2: The consumer group has the same number of consumers as the number of partitions in a topic.

If we add 2 consumers to Consumer Group 1, making the number of consumers equal to the number of partitions, each consumer will read events from one corresponding partition. In cases where consumers perform high-latency operations such as writing to a database or performing time-consuming computations on the data, increasing the number of consumers will distribute the load, thereby speeding up the data reading process from a topic.

• Scenario 3: The consumer group has more consumers than the number of partitions in a topic.

We should not have more consumers than partitions in a topic because some consumers may become idle as all partitions are occupied, leading to events being missed or not read.

About producers in Apache Kafka

Kafka Producer is a client application that publishes events to a specific topic in Kafka and always writes to the leader broker. By default, producers do not care which partition an event is written to and will publish events evenly across all partitions of a topic. In some cases, a producer will send events directly to specific partitions.

Producers connect to Kafka Brokers via the TCP network protocol. This is a bi-directional connection.

The diagram below provides an overview of the components in Kafka producers.

The process of sending events from Kafka producers to Kafka brokers involves 4 steps.

• Step 1: Create a ProducerRecord

A producer publishes an event to Kafka by creating a ProducerRecord, which must include a topic and value, and optionally a partition and key.

• Step 2: Serializer

Before sending the ProducerRecord over the network, the producer will serialize the key and value into ByteArrays (byte arrays).

• Step 3: Determine the number of partitions

After the Serializer step, the data is sent to a Partitioner. If a partition is specified, the Partitioner returns the specified partition; otherwise, the Partitioner selects a partition based on the ProducerRecord key.

• Step 4: Brokers process the events and return results to the producers

Once the producer knows which topic and partition the event needs to be sent to, it adds the event to a batch of records. These records will then be sent together to the same topic and partition. An independent thread is responsible for sending these batches of records to the appropriate Kafka brokers.

The broker sends back a response upon receiving the events. If the event is successfully written to Kafka, the broker returns an object containing RecordMetadata, which includes the topic, partition, and offset of the record within the partition. If the write is unsuccessful, the broker returns an error. When the producer receives an error, the event will be retried several times before giving up and returning an error.

About Apache Kafka topics

Topics classify messages in Kafka. A topic is roughly analogous to a database table or folder. Topics are further subdivided into several partitions as described above

A single topic can be scaled horizontally across various servers to deliver performance well beyond the capabilities of a single server. A Kafka cluster sustains the partitioned log for each topic. Note that while a topic generally has multiple partitions, there is no assurance of message time-ordering throughout the whole topic – only within a single partition.

About Apache Kafka Zookeeper

Zookeeper stores metadata for the Kafka brokers. It acts as a liaison between the broker and the consumers, enabling distributed processes to communicate with one another via a common centralized namespace of data registers called znodes.

“ZooKeeper is a distributed, open-source coordination service for distributed applications.”

Source: ZooKeeper home page

Note

When it comes to production, a recommended docker image tagging method is semantic versioning (Semver).

With the introduction of Apache Kafka 3.0, Zookeeper is in the process of being removed. Many users had complained about having to manage a separate system and the single point of failure that Zookeeper created. Going forward, Kafka brokers will essentially assume Zookeeper’s functions, storing metadata locally in a file. The Controller takes over registering brokers and removing failed brokers from the cluster, and upon startup the brokers just read from the Controller what has changed, not the full state. This enables Kafka to support more partitions with less CPU consumption.

About queues in Apache Kafka

Initially Kafka was developed as a messaging queue. But it took on a life of its own, and as a result, was donated to Apache for further development. Kafka still operates like a traditional messaging queue such as RabbitMQ, in that it enables you to publish and subscribe to streams of messages. But comparing RabbitMQ vs Kafka, there are three core differences.

1. Kafka operates as a modern distributed system that runs as a cluster and can scale to handle any number of applications.

2. Kafka is designed to serve as a storage system and can store data as long as necessary – most message queues (like RabbitMQ) remove messages immediately after the consumer confirms receipt.

3. Kafka is designed to handle stream processing, computing derived streams and datasets dynamically, rather than just in batches of messages.

About the Apache Kafka schema registry

A schema registry handles message schemas and routes them to topics. The schema registry verifies and maintains schemas implemented for Kafka messages. It also imposes compatibility before message addition. In this way publishers know which topics receive which forms (schemas) of events, and subscribers know how to interpret and extract information from events in a topic. The schema registry sends data serialized per schema ID so the consumer can map a schema to a message type.

Kafka stores the schema’s namespace in the record it receives from publishers. Then the reader uses the namespace to retrieve records from the schema registry and deserialize the data.

The schema registry saves Kafka from enumerating the same schema details repeatedly into every message. Producers communicate with the registry and write new message schema to it. The producer obtains the schema ID from the registry and includes that in the message.

About Core Kafka Enhancements

Beyond the core Kafka architecture are the following enhancements that extend the capabilities of Kafka. Kafka Streams integrates stream processing with Kafka and Kafka Connect facilitates connecting Kafka to external data sources and sinks.

What is Kafka Streams?

Kafka Streams is an API-driven client library for building applications and microservices, where the input and output data are stored in Kafka clusters. Kafka Streams jobs do not execute on Kafka brokers; rather, they run inside your application or microservice.

Kafka Streams’ approach to parallel processing resembles that of Kafka:

• Each stream partition is a fully-ordered sequence of data records that maps to a Kafka topic partition.

• A data record in the stream maps to a Kafka message from that topic.

• The keys of data records determine the partitioning of data in both Kafka and Kafka Streams – that is, how data is routed to specific partitions within topics.

Kafka Streams processes messages in real-time (that is, not in microbatches), with millisecond latency. It supports stateless and stateful processing and window operations.

Kafka Streams includes two special processors: a Source Processor and a Sink Processor.

• The source processor produces an input stream by consuming records from one or more Kafka topics and forwarding them to its down-stream processors.

• Sink Processor: A sink processor sends any received records from its up-stream processors to a specified Kafka topic.

The processed results can either be streamed back into Kafka or written to an external system.

2. What are Kafka Connect and Kafka Connectors?

Kafka Connect provides a scalable means of moving data between Kafka and other repositories. It provides APIs and a runtime for designing and operating connector plugins, which are frameworks that Kafka Connect executes and are essential for data movement.

The connector controls:

• The definition and number of tasks that operate for the connector

• How to divide the data-copying work among tasks.

• Getting task configurations from workers and passing them on to the target.

3. What are Kafka APIs?

Kafka provides five key APIs for Java, Scala, and command-line tools for management and administrative tasks. In addition, these APIs enable you to communicate with Kafka programmatically:

• The Admin API enables you to manage and analyze Kafka topics, brokers, Access Control Lists, and other objects.

• The Producer API enables applications to submit (write) data streams to Kafka cluster topics.

• The Consumer API enables programs to access (read) data streams from topics in the Kafka cluster.

• The Streams API provides stream processes that convert a data stream into an output stream.

The Connect API creates connections that continuously draw data from a source data system into Kafka or push data from Kafka into a target data system. (The Connect API is not often needed, as you can instead use pre-built connections without writing any code.)

Advantages of Kafka

• Scalability: The distributed architecture of Kafka enables seamless scalability. It can handle large volumes of data and increase data throughput by adding more brokers and partitions.

• Durability: Kafka ensures data durability through replication. Data is replicated across multiple brokers, preventing data loss even if brokers fail. Additionally, by adhering to the Append-only commit log principle, Kafka's commit log provides data durability, fault tolerance, and efficient message processing.

• Real-time processing: Kafka enables real-time data transmission and processing, making it suitable for applications that require low-latency data analysis and transmission.

• Open-source: Kafka is open-source software, providing the freedom to modify, customize, and extend its functionality to meet specific requirements. It can be used without paying licensing fees, helping to reduce costs associated with proprietary software. Kafka is cross-platform and can integrate with various tools and libraries, creating a large ecosystem.

• Long polling:: Although long polling is not a built-in feature of Kafka, the concept can be applied to how Kafka consumers interact with Kafka brokers to achieve efficient data consumption.

Disadvantages of Kafka

• ZooKeepers dependency: In older versions of Kafka, there was a dependency on Apache ZooKeeper for cluster coordination, which added complexity and contained potential points of failure.

• Complexity: The distributed nature of Kafka can introduce complexity in setup, configuration, and management, requiring expertise for effective deployment and maintenance. Additionally, learning to use Kafka and its ecosystem components effectively can take time and effort, especially for users new to distributed systems.

• Resource requirement: Managing Kafka clusters requires significant hardware resources, including memory, storage capacity, and network bandwidth.

Important

Kafka Architecture: Key Takeaways

• Apache Kafka is an open-source stream-processing software platform that helps deliver real-time data feeds to applications.

• Kafka stores data durably, distributes it across a cluster of nodes, and replicates partitions and replicas to ensure data consistency and resilience to failures.

• Kafka provides log aggregation, stream processing, commit logs, and clickstream tracking.

• Kafka is comprised of brokers, controllers, partitions, consumers, producers, topics, schema registries, and Zookeeper (phasing out).

• Consumers and producers interact with topics, and Kafka Streams provides stream processing.

Conclusion

Apache Kafka has redefined how organizations handle data streaming and event processing. Its distributed architecture, fault tolerance, and scalability make Kafka a powerful tool for managing the challenges of real-time data. From collecting and analyzing user interactions to building event-driven applications, Kafka's capabilities are vast and impactful.