Kafka is a stream processing platform which groups like-messages together, such that the number of sequential writes to disk can be increased, thus effectively increasing database speeds.
The message-bus-kafka package consists of a few components, those being Kafka, Kafdrop, and Kminion.
The services consuming from and producing to kafka might crash if Kafka is unreachable, so this is something to bear in mind when making changes to or restarting the kafka service.
The core stream-processing element of the message-bus-kafka package.
Kafdrop is a web user-interface for viewing Kafka topics and browsing consumer-groups.
A prometheus exporter for Kafka.
Using a config importer, Kafka's topics are imported to Kafka. The topics are specified using the KAFKA_TOPICS environment variable, and must be of syntax:
topic or topic:partition:replicationFactor
Using topics 2xx, 3xx, and metrics (partition=3, replicationFactor=1) as an example, we would declare:
KAFKA_TOPICS=2xx,3xx,metrics:3:1
where topics are separated by commas.
Kafdrop - http://127.0.0.1:9013/
Listed in this page are all environment variables needed to run the Message Bus Kafka.
KAFKA_INSTANCES
Number
Service replicas
No
1
KAFKA_CPU_LIMIT
Number
CPU usage limit
No
0
KAFKA_CPU_RESERVE
Number
Reserved CPU
No
0.05
KAFKA_MEMORY_LIMIT
String
RAM usage limit
No
3G
KAFKA_MEMORY_RESERVE
String
Reserved RAM
No
500M
KAFKA_TOPICS
String
Kafka topics
Yes
ZOOKEEPER_CPU_LIMIT
Number
CPU usage limit
No
0
ZOOKEEPER_CPU_RESERVE
Number
Reserved CPU
No
0.05
ZOOKEEPER_MEMORY_LIMIT
String
RAM usage limit
No
3G
ZOOKEEPER_MEMORY_RESERVE
String
Reserved RAM
No
500M
KMINION_CPU_LIMIT
Number
CPU usage limit
No
0
KMINION_CPU_RESERVE
Number
Reserved CPU
No
0.05
KMINION_MEMORY_LIMIT
String
RAM usage limit
No
3G
KMINION_MEMORY_RESERVE
String
Reserved RAM
No
500M
KAFDROP_CPU_LIMIT
Number
CPU usage limit
No
0
KAFDROP_CPU_RESERVE
Number
Reserved CPU
No
0.05
KAFDROP_MEMORY_LIMIT
String
RAM usage limit
No
3G
KAFDROP_MEMORY_RESERVE
String
Reserved RAM
No
500M