One of the clients was performing a study and needed help identifying the different factors that would influence the decision made by small medium or large enterprises in choosing between the different options available for kafka.
I did the research and categorized the needs based on whats a must have for small business vs for a large enterprise and which of the available options for kafka provide that.
I have summarized some of the costs for each of the options for equivalent implementations. As always reach out to the Data Experts at Aqib Technologies if you have questions / concerns / comments or need help in any implementation.
| Features | Small/Medium business | Large Enterprise | Apache Kafka | Confluent Kafka | AWS Kinesis | AWS MSK |
| Serverless | Critical | Not needed | N | Y | Y | Y |
| Elastic Scaling | Critical | Nice to have | N | Y | Y | Y |
| Infinite Storage / Tiered Storage | Nice to have | Nice to have | N | Y | N | Y |
| High Availability | Critical | Critical | Y | Y | Y | Y |
| No ZooKeeper management | Nice to have | Nice to have | Y | Y | Y | N |
| No-touch patching and upgrades | Nice to have | Critical | N | Y | Y | Y |
| Connect to data sources / data sinks | Critical | Critical | Y | Y | Y | Y |
| Kafka Streams | Critical | Critical | Y | Y | Y | Y |
| Java Clients | Critical | Critical | Y | Y | Y | Y |
| Non Java Clients | Critical | Critical | Y | Y | Y | Y |
| REST Proxy | Critical | Critical | Y | Y | Y | Y |
| Connectors | Critical | Critical | Y | Y | Y | Y |
| MQTT Proxy | Critical | Critical | Y | Y | Y | Y |
| Schema Registry | Nice to have | Nice to have | Y | Y | Y | Y |
| ksqlDB | Not needed | Not needed | N | Y | Y | N |
| Cloud UI / Control Center | Critical | Nice to have | N | Y | Y | Y |
| Metrics API | Critical | Critical | Y | Y | Y | Y |
| Datadog / Prometheus Integration | Nice to have | Nice to have | N | Y | Y | Y |
| Health+ | Critical | Critical | N | Y | Y | Y |
| Admin REST APIs | Nice to have | Nice to have | N | Y | Y | Y |
| Confluent for Kubernetes | Nice to have | Nice to have | N | Y | Y | Y |
| Ansible Playbooks | Nice to have | Nice to have | N | Y | Y | Y |
| Self-Balancing Clusters | Nice to have | Critical | N | Y | N | Y |
| Access Control Lists | Nice to have | Nice to have | N | Y | Y | Y |
| Role-Based Access Control | Nice to have | Critical | N | Y | Y | Y |
| Audit Logs | Nice to have | Critical | N | Y | Y | Y |
| Secret Protection / Encryption | Nice to have | Critical | N | Y | Y | Y |
| Bring Your Own Key | Nice to have | Critical | N | Y | Y | Y |
| Private Networking | Nice to have | Critical | N | Y | Y | Y |
| HIPAA/GDPR/CCPA readiness | Not needed | Critical | N | Y | Y | Y |
| Schema Validation | Nice to have | Critical | N | Y | Y | Y |
| Stream Catalog | Nice to have | Nice to have | N | Y | Y | Y |
| Stream Lineage | Nice to have | Nice to have | N | Y | Y | Y |
| 99.99% Uptime SLA | Nice to have | Critical | N | Y | Y | Y |
| Cluster Linking | Nice to have | Nice to have | N | Y | Y | Y |
| Multi-AZ / Multi-Region Clusters | Nice to have | Critical | N | Y | Y | Y |
| Replication | Critical | Critical | Y | Y | Y | Y |
| 24x7x365 Expert Support | Critical | Nice to have | N | Y | Y | Y |
| Professional Services | Critical | Nice to have | N | Y | Y | Y |
| Education | Critical | Critical | Y | Y | Y | Y |
| Monthly Cost in Dollars for 1 MB/Sec | 60 | 750 | 900 | 300 | ||
| Performance (MB ./ sec ) | 600 | 600 | 200 | 600 | ||
| Ease of Deployment (Scale 1 – 5) | Critical | Critical | 5 | 1 | 2 | 2 |
| Ease of Maintenance (Scale 1 – 5) | Critical | Critical | 2 | 1 | 1 | 1 |