MQTT to Kafka: why both exist and how to connect them
MQTT and Apache Kafka solve different problems: MQTT connects many constrained devices over unreliable networks, while Kafka stores durable, replayable streams for analytics and backend services. Most IoT architectures use both, with MQTT at the edge and Kafka behind it, connected through Kafka Connect, a broker-side bridge or a custom consumer service.
Why use both MQTT and Kafka?
MQTT and Kafka are used together because they solve different problems at different points in the data path. MQTT is a lightweight publish/subscribe protocol for getting data off many devices over unreliable networks. Apache Kafka is a distributed log for storing streams durably and letting many backend systems read and replay them.
A typical industrial or IoT architecture uses MQTT at the edge and Kafka in the data centre or cloud. Machines and gateways publish to an MQTT broker. A bridge moves the data into Kafka. Analytics, data lakes, stream processing and business applications consume from Kafka.
MQTT vs Kafka: different roles
The two technologies are often compared, but they fill different roles.
| Aspect | MQTT | Apache Kafka |
|---|---|---|
| Main purpose | Device connectivity and message delivery | Durable, replayable event streams |
| Typical clients | Sensors, PLC gateways, vehicles, apps | Backend services, stream processors, databases |
| Number of clients | Very many, often on weak links | Fewer, in stable networks |
| Topics | Hierarchical, fine-grained, created on the fly | Flat, coarse, usually created up front |
| Storage | Delivery queues and one retained message per topic | Log retained by time or size, readable again |
| Consumer scaling | Shared subscriptions (MQTT 5) | Consumer groups over partitions |
| Ordering | Per client and topic, within a session | Per partition |
| Delivery control | QoS 0, 1, 2 per hop | Producer acks, idempotence, transactions |
| Disconnect handling | Keep alive, Last Will, persistent sessions | Not designed for frequently disconnecting clients |
MQTT is strong where the network is the problem. Kafka is strong where storage, replay and fan-out to many backend consumers are the problem.
Integration patterns
There are three common ways to move data from an MQTT broker into Kafka.
Kafka Connect with an MQTT connector
Kafka Connect is Kafka’s framework for moving data in and out of Kafka. An MQTT source connector subscribes to MQTT topic filters and writes records to Kafka. An MQTT sink connector does the reverse and publishes Kafka records to MQTT. This pattern needs no custom code and uses Connect’s scaling and offset handling. Check which MQTT versions, QoS levels and mapping options the connector you choose supports.
Broker-side bridge or rule engine
Some MQTT brokers can forward messages to Kafka directly, through a built-in bridge, plugin or rule engine. This avoids a separate service and keeps the data path short. The trade-off is that integration logic lives in the broker configuration.
Custom consumer service
A small service subscribes to MQTT and produces to Kafka using standard client libraries. This gives full control over filtering, payload transformation, enrichment and error handling. You also own its deployment, monitoring and scaling. With MQTT 5, several instances can split the load with a shared subscription:
$share/kafka-bridge/plant/+/temp
Mapping topics and choosing partition keys
Map many fine-grained MQTT topics to a few Kafka topics, and keep the detail in the key or headers. MQTT topics such as enterprise/site-frankfurt/line-3/press-07/temp can number in the thousands or millions. Creating one Kafka topic per MQTT topic does not scale well, because every Kafka topic and partition adds overhead to the cluster.
Common mappings:
- By data type: all temperature readings go to
telemetry.temperature. - By site or area: everything from one site goes to
site-frankfurt.telemetry. - One topic for all telemetry, with the original MQTT topic in a record header.
The record key decides the partition. Kafka’s default partitioner sends records with the same key to the same partition, and Kafka guarantees order only within a partition. Using the device ID, or the full MQTT topic, as the key keeps each device’s messages in order. Watch for skew: if a few devices produce most of the traffic, their partitions become hot.
Delivery guarantees: QoS vs Kafka semantics
MQTT QoS and Kafka delivery settings each cover one hop, so the end-to-end guarantee is only as strong as the weakest link. MQTT QoS 1 and 2 apply between a publisher and the broker, and separately between the broker and each subscriber. Kafka’s producer settings (acks=all, idempotence) apply between the producer and the Kafka cluster.
For a reliable pipeline:
- Publish with QoS 1 from devices where data loss matters.
- Let the bridge subscribe with QoS 1 and a persistent session (MQTT 5: Clean Start off and a Session Expiry Interval long enough to cover restarts), so messages queue while the bridge is down.
- Acknowledge the MQTT message only after Kafka has confirmed the write, if your bridge allows it.
- Expect duplicates after failures and make consumers idempotent.
In practice this gives at-least-once delivery from device to Kafka. Kafka’s exactly-once features apply to processing within Kafka, not to the MQTT hop.
Handling back-pressure
Back-pressure appears when Kafka accepts data more slowly than MQTT delivers it. If the bridge cannot write to Kafka, for example during a cluster outage, messages build up somewhere. Decide where:
- In the MQTT broker: queued messages for the bridge’s persistent session. Set limits so the broker’s memory or disk is not exhausted.
- In the bridge: a bounded in-memory or on-disk buffer. Producer buffers in the Kafka client are bounded too, and the bridge must handle the case when they are full.
- At the edge: gateways that buffer locally and forward later.
MQTT 5’s Receive Maximum limits how many unacknowledged QoS 1 and 2 messages the broker sends to the bridge at once. Combined with acknowledging only after a successful Kafka write, this slows delivery from the broker instead of losing data in the bridge. Monitor queue depth, consumer lag and bridge throughput so you see problems before buffers fill up.