kafka vs HTTP (2024)

When to prefer kafka and why…..

Why to use Kafka?

Kafka combines three key capabilities so you can implement your use cases for event streaming end-to-end with a single battle-tested solution:

1. To publish (write) and subscribe to (read) streams of events, including continuous import/export of your data from other systems.
2. To store streams of events durably and reliably for as long as you want.
3. To process streams of events as they occur or retrospectively.

1. Kafka — Publish & Subscribe (just process the pipeline, will notify once the job is done)
   REST — Request & Await response (on-demand)
2. Kafka — Publish once — Subscribe n times (by n components).
   REST — Request once, get the response once. Deal over.
3. Kafka — Data is stored in topic. Seek back & forth (offsets) whenever you want till the topic is retained.
   REST — Once the response is over, it is over. Manually employ a database to store the processed data.
4. Kafka — Split the processing, have intermediate data stored in intermediate topics (for speed and fault-tolerance)
   REST — Take the data, process it all at once OR if you wish to break it down, don’t forget to take care of your OWN intermediate data stores.
5. Kafka — The one who makes the request typically is not interested in a response (except the response that if the message is sent)
   REST — I am making the request means I typically expect a response (not just a response that you have received the request, but something that is meaningful to me, some computed result for example!)
6. Kafka — Doesn’t support request/response model
   REST — supports it

1. Communication Style:

• Kafka: Kafka follows a publish-subscribe model, where producers send messages to topics, and consumers subscribe to these topics to receive messages asynchronously. It is suitable for scenarios where events need to be broadcast to multiple consumers.
• REST: REST is request-response based. Clients make HTTP requests (GET, POST, PUT, DELETE, etc.) to interact with resources on the server, and the server responds with the requested data or performs the requested operation.

2. Latency:

• Kafka: Kafka is optimized for low-latency event streaming and can provide real-time data processing capabilities.
• REST: REST can introduce higher latency, especially when dealing with complex requests or when a large number of requests are made due to its request-response nature.

3. Data Structure:

• Kafka: Kafka deals with structured or semi-structured data in the form of messages or events. Messages can be of various formats (JSON, Avro, etc.).
• REST: REST typically deals with structured data in the form of resources, often represented in JSON or XML format.

4. Scalability:

• Kafka: Kafka is highly scalable and can handle large volumes of data and high-throughput scenarios by distributing data across multiple partitions and brokers.
• REST: Scalability of REST APIs depends on the server implementation. It may require load balancing and other mechanisms to handle high loads.

5. Use Cases:

• Kafka: Kafka is suitable for use cases such as real-time data processing, event sourcing, log aggregation, and building event-driven microservices.
• REST: REST is commonly used for building web services and APIs for web and mobile applications, where CRUD operations and resource management are essential.

6. Complexity:

• Kafka: Implementing Kafka-based solutions can be more complex and may require a deeper understanding of event-driven architectures.
• REST: RESTful APIs are typically easier to understand and implement, making them a common choice for web-based applications especially for CRUD operations only.

Now I think you know what is micro-service and why to use KAFKA

Refrence

• Request-Response with REST/HTTP vs. Data Streaming with Apache Kafka — Friends, Enemies, Frenemies? — Kai Waehner (kai-waehner.de)

In upcoming deck — case study of KAFKA vs REST

My LinkedIn profile for any queries : https://in.linkedin.com/in/vikalprusia