1 of 3

Message Bus - Kafka

Kafka is a stream processing platform which groups like-messages together, such that the number of sequential writes to disk can be increased, thus effectively increasing database speeds.

Components

The message-bus-kafka package consists of a few components, those being Kafka, Kafdrop, and Kminion.

The services consuming from and producing to kafka might crash if Kafka is unreachable, so this is something to bear in mind when making changes to or restarting the kafka service.

Kafka

The core stream-processing element of the message-bus-kafka package.

Kafdrop

Kafdrop is a web user-interface for viewing Kafka topics and browsing consumer-groups.

Kminion

A prometheus exporter for Kafka.

Local Development

Kafka Topics Configuration

Using a config importer, Kafka's topics are imported to Kafka. The topics are specified using the KAFKA_TOPICS environment variable, and must be of syntax:

topic or topic:partition:replicationFactor

Using topics 2xx, 3xx, and metrics (partition=3, replicationFactor=1) as an example, we would declare:

KAFKA_TOPICS=2xx,3xx,metrics:3:1

where topics are separated by commas.

Accessing Kafdrop

Kafdrop -

Environment Variables

Listed in this page are all environment variables needed to run the Message Bus Kafka.

Variable Name

Type

Relevance

Required

Default

KAFKA_INSTANCES

Number

Service replicas

Local Development

Kafka Topics Configuration

Using a config importer, Kafka's topics are imported to Kafka. The topics are specified using the KAFKA_TOPICS environment variable, and must be of syntax:

topic or topic:partition:replicationFactor

Using topics 2xx, 3xx, and metrics (partition=3, replicationFactor=1) as an example, we would declare:

KAFKA_TOPICS=2xx,3xx,metrics:3:1

where topics are separated by commas.

Accessing Kafdrop

Kafdrop -

Message Bus - Kafka

Kafka is a stream processing platform which groups like-messages together, such that the number of sequential writes to disk can be increased, thus effectively increasing database speeds.

Components

The message-bus-kafka package consists of a few components, those being Kafka, Kafdrop, and Kminion.

The services consuming from and producing to kafka might crash if Kafka is unreachable, so this is something to bear in mind when making changes to or restarting the kafka service.

Kafka

The core stream-processing element of the message-bus-kafka package.

Kafdrop

Kafdrop is a web user-interface for viewing Kafka topics and browsing consumer-groups.

Kminion

A prometheus exporter for Kafka.

Message Bus - Kafka

hashtagComponents

hashtagKafka

hashtagKafdrop

hashtagKminion

Local Development

hashtagKafka Topics Configuration

hashtagAccessing Kafdrop

Environment Variables

Environment Variables

Local Development

hashtagKafka Topics Configuration

hashtagAccessing Kafdrop

Message Bus - Kafka

hashtagComponents

hashtagKafka

hashtagKafdrop

hashtagKminion

Components

Kafka

Kafdrop

Kminion

Kafka Topics Configuration

Accessing Kafdrop

Kafka Topics Configuration

Accessing Kafdrop

Components

Kafka

Kafdrop

Kminion