Comparison: Fluvio vs Kafka

This purpose of this document is to compare and contrast Apache Kafka with Fluvio Open Source.

Fluvio is a cloud native data streaming runtime that was architected from the ground up in Rust to solve the challenges with Kafka and alternatives to Kafka.

Fluvio is built to deliver a lean and mean data streaming platform with all the core functionality with unmatched performance and efficiency without the complexities and overhead of operating and managing Kafka.

Apache Kafka is a ubiquitous open-source streaming platform used for real-time data processing and event-driven architectures. It allows data to be published to topics and consumed by various applications, making it popular for building scalable and fault-tolerant dataflows.

According to the official website, Apache Kafka is widely used by 80% of Fortune 100 companies as a robust distributed log solution.

At the same time if we look at the Confluent or the RedPanda website it is clear that there are several challenges with Kafka in terms of performance, efficiency, complexity, maintenance toil, ops overhead, and costs.

There are 2 types of alternatives available to solve the Kafka challenges.

• Managed cloud native solutions based on Kafka
• Alternative streaming platforms compatible with Kafka

InfinyOn Fluvio emerges as a cutting-edge streaming platform, designed to seamlessly integrate with cloud-native architectures. It distinguishes itself through a user-centric interface and the incorporation of SmartModules for in-stream data transformations, achieving unparalleled resource efficiency via Rust and WebAssembly.

Unlike alternative streaming platforms, Fluvio is a single small binary that consumes minimal resources to run and operates using a simple and powerful client and CLI to build streaming flows without the limitations of the JVM.

Fluvio streaming runtime is complemented with SDKs to develop protocol level connectors and web assembly based on stream transformations that offers fine grained composability and control, and is also immune to memory issues.

To run kafka cluster locally, need to download images amounting to 6.97 GB:

• enterprise-control-center - 1.31 GB
• enterprise-kafka - 1.01 GB
• kafka-rest - 1.85 GB
• schema-registry - 1.94 GB
• zookeeper - 863 MB.

Fluvio is a single binary less than 150 Mb that currently has a dependency on Kubernetes that we are decoupling to get a single binary.

While we are working on our implementation of the open messaging benchmark, one of our customers compared their kafka clusters with fluvio and shared their results with us.

As per our customer data, Fluvio turned out to be 7 times leaner in CPU utilization, consumed 50 times less memory, while delivering 5 times more throughput.

Parameter	Infinyon Fluvio	Apache Kafka
License	Apache 2.0	Apache 2.0
Components	Fluvio + Kubernetes (K8 decoupling ready by Q4 2023 to support single binary deployment)	Kafka + Zookeeper/KRaft (KIP-833)
Message Consumption Model	Pull	Pull
Storage Architecture	Log	Log

Both Fluvio and Kafka use Apache 2.0, which is a fully open-source license.

Kafka uses Apache Zookeeper in production for its distributed coordination and configuration management, until KRaft mode is standard.

Fluvio uses Kubernetes for its cluster and configuration management on InfinyOn Cloud. For Fluvio open source we are enabling deploying local clusters with a single independent binary.

Both Kafka and Fluvio use pull-based message consumption models. Suitable for event streaming use-cases.

Both Kafka and Fluvio use a storage layer built on a distributed log. New entries are added to the end of the log, and data is read in a sequential fashion starting from an offset. Data is transferred using zero-copy techniques to move from the disk buffer to the network buffer.

Using Fluvio SmartModules, data can be transformed in-stream. For example, before it is written to the log, or before it is sent over the network. This can favorably reduce bandwidth requirements and eliminate time spent cleaning data afterwards.

Parameter	Infinyon Fluvio	Apache Kafka
Deployment	Run 1 CLI command	Requires experience for tuning performance
Enterprise Support	InfinyOn Cloud	3rd parties like Confluent, AWS MSK
Managed Cloud Offerings	InfinyOn Cloud	3rd parties like Confluent, AWS MSK
Self-Healing	Yes	No

Kafka is highly configurable, and may require time investment for performance. Production installations have zookeeper to maintain until KIP-833.

Fluvio requires a binary installation and a Kubernetes cluster. A cluster can be deployed with a single CLI command

Kafka is offered by many vendors for enterprise support, and managed cloud offerings.

InfinyOn provides enterprise support, as well as a managed cloud offering InfinyOn Cloud

Kafka clusters can require an operator to invest in the skills or services for performance tuning, management, and monitoring.

Fluvio clusters come out of the box with internal cluster management with Kubernetes operators.

Parameter	InfinyOn Fluvio	Apache Kafka
Replication	Yes	Yes
Multi-tenancy	Yes via k8 namespaces	No
Ordering guarantees	Partition Level	Partition level
Permanent storage	Yes	Yes
Delivery guarantees	At least once, Exactly once	At least once, Exactly once
Idempotency	Yes	Yes
Geo-Replication (Multi-region)	Yes - Mirror topic	Yes

Kafka and Fluvio replication work by having multiple copies spread over multiple brokers/SPUs for High Availability storage.

Kafka doesn’t natively support multi-tenancy, but it can be achieved.

Fluvio supports multi-tenancy through Kubernetes namespaces.

Both Kafka and Fluvio have partition level ordering guarantees.

Both Kafka and Fluvio have durable, reliable storage. Data retention is configured at the topic level.

Both Kafka and Fluvio support At least once, and Exactly Once.

Kafka supports idempotent producers, but clients are responsible for retrying transmission.

Fluvio SmartModules can implement idempotent producers and consumers with its deduplication filter.

Kafka MirrorMaker supports data replication between different Kafka environments

Fluvio topics can be configured to mirror different Fluvio environments

Parameter	Infinyon Fluvio	Apache Kafka
GUI	Yes w/ InfinyOn Cloud	3rd party
Schema Management	No (In development)	No
Message Routing	Partial	Yes. Using Kafka connect and KStreams
Log Compaction	No	Yes
Message Replay, time travel	Yes	Yes
Stream enrichment	SmartModules	SQL-based using KStreams
Pull retry mechanism	Client responsibility	Client responsibility

Kafka GUIs exist and are supported by 3rd-parties

Fluvio does not have a native GUI. InfinyOn Cloud serves as a GUI for Fluvio

Kafka has Schema Registry which is offered by Confluent.

Fluvio does not have schema management yet, but is planned.

Kafka Connect and KStreams can dynamically route events other topics based on data and also branch a single event stream into multiple new topics

Fluvio Connectors with SmartModules can filter, transform or enrich messages to downstream services. Fluvio does not support dynamic routing of messages to other topics.

Kafka supports log compaction to retain the last known value for each message key for a single topic partition.

Fluvio does not support log compaction

Kafka and Fluvio both support the ability to replay messages by consumers seeking specific offsets

Kafka Streams and Connect can provide stream enrichment

Fluvio SmartModules provide stream enrichment

In both Kafka, and Fluvio, the client is responsible for implementing a retry mechanism based on their needs.