learn/mqtt

MQTT to Kafka: why both exist and how to connect them

Short answer

MQTT and Apache Kafka solve different problems: MQTT connects many constrained devices over unreliable networks, while Kafka stores durable, replayable streams for analytics and backend services. Most IoT architectures use both, with MQTT at the edge and Kafka behind it, connected through Kafka Connect, a broker-side bridge or a custom consumer service.

Updated:

Why use both MQTT and Kafka?

MQTT and Kafka are used together because they solve different problems at different points in the data path. MQTT is a lightweight publish/subscribe protocol for getting data off many devices over unreliable networks. Apache Kafka is a distributed log for storing streams durably and letting many backend systems read and replay them.

A typical industrial or IoT architecture uses MQTT at the edge and Kafka in the data centre or cloud. Machines and gateways publish to an MQTT broker. A bridge moves the data into Kafka. Analytics, data lakes, stream processing and business applications consume from Kafka.

MQTT vs Kafka: different roles

The two technologies are often compared, but they fill different roles.

Aspect MQTT Apache Kafka
Main purpose Device connectivity and message delivery Durable, replayable event streams
Typical clients Sensors, PLC gateways, vehicles, apps Backend services, stream processors, databases
Number of clients Very many, often on weak links Fewer, in stable networks
Topics Hierarchical, fine-grained, created on the fly Flat, coarse, usually created up front
Storage Delivery queues and one retained message per topic Log retained by time or size, readable again
Consumer scaling Shared subscriptions (MQTT 5) Consumer groups over partitions
Ordering Per client and topic, within a session Per partition
Delivery control QoS 0, 1, 2 per hop Producer acks, idempotence, transactions
Disconnect handling Keep alive, Last Will, persistent sessions Not designed for frequently disconnecting clients

MQTT is strong where the network is the problem. Kafka is strong where storage, replay and fan-out to many backend consumers are the problem.

Integration patterns

There are three common ways to move data from an MQTT broker into Kafka.

Kafka Connect with an MQTT connector

Kafka Connect is Kafka’s framework for moving data in and out of Kafka. An MQTT source connector subscribes to MQTT topic filters and writes records to Kafka. An MQTT sink connector does the reverse and publishes Kafka records to MQTT. This pattern needs no custom code and uses Connect’s scaling and offset handling. Check which MQTT versions, QoS levels and mapping options the connector you choose supports.

Broker-side bridge or rule engine

Some MQTT brokers can forward messages to Kafka directly, through a built-in bridge, plugin or rule engine. This avoids a separate service and keeps the data path short. The trade-off is that integration logic lives in the broker configuration.

Custom consumer service

A small service subscribes to MQTT and produces to Kafka using standard client libraries. This gives full control over filtering, payload transformation, enrichment and error handling. You also own its deployment, monitoring and scaling. With MQTT 5, several instances can split the load with a shared subscription:

$share/kafka-bridge/plant/+/temp

Mapping topics and choosing partition keys

Map many fine-grained MQTT topics to a few Kafka topics, and keep the detail in the key or headers. MQTT topics such as enterprise/site-frankfurt/line-3/press-07/temp can number in the thousands or millions. Creating one Kafka topic per MQTT topic does not scale well, because every Kafka topic and partition adds overhead to the cluster.

Common mappings:

  • By data type: all temperature readings go to telemetry.temperature.
  • By site or area: everything from one site goes to site-frankfurt.telemetry.
  • One topic for all telemetry, with the original MQTT topic in a record header.

The record key decides the partition. Kafka’s default partitioner sends records with the same key to the same partition, and Kafka guarantees order only within a partition. Using the device ID, or the full MQTT topic, as the key keeps each device’s messages in order. Watch for skew: if a few devices produce most of the traffic, their partitions become hot.

Delivery guarantees: QoS vs Kafka semantics

MQTT QoS and Kafka delivery settings each cover one hop, so the end-to-end guarantee is only as strong as the weakest link. MQTT QoS 1 and 2 apply between a publisher and the broker, and separately between the broker and each subscriber. Kafka’s producer settings (acks=all, idempotence) apply between the producer and the Kafka cluster.

For a reliable pipeline:

  • Publish with QoS 1 from devices where data loss matters.
  • Let the bridge subscribe with QoS 1 and a persistent session (MQTT 5: Clean Start off and a Session Expiry Interval long enough to cover restarts), so messages queue while the bridge is down.
  • Acknowledge the MQTT message only after Kafka has confirmed the write, if your bridge allows it.
  • Expect duplicates after failures and make consumers idempotent.

In practice this gives at-least-once delivery from device to Kafka. Kafka’s exactly-once features apply to processing within Kafka, not to the MQTT hop.

Handling back-pressure

Back-pressure appears when Kafka accepts data more slowly than MQTT delivers it. If the bridge cannot write to Kafka, for example during a cluster outage, messages build up somewhere. Decide where:

  • In the MQTT broker: queued messages for the bridge’s persistent session. Set limits so the broker’s memory or disk is not exhausted.
  • In the bridge: a bounded in-memory or on-disk buffer. Producer buffers in the Kafka client are bounded too, and the bridge must handle the case when they are full.
  • At the edge: gateways that buffer locally and forward later.

MQTT 5’s Receive Maximum limits how many unacknowledged QoS 1 and 2 messages the broker sends to the bridge at once. Combined with acknowledging only after a successful Kafka write, this slows delivery from the broker instead of losing data in the bridge. Monitor queue depth, consumer lag and bridge throughput so you see problems before buffers fill up.

01ultramq/learn/faq

Frequently asked questions

Can Kafka replace an MQTT broker?

Usually not for devices. Kafka clients are heavier and expect stable connections to the cluster, while MQTT is designed for many small devices on unreliable networks. Kafka is typically placed behind an MQTT broker.

How do MQTT topics map to Kafka topics?

MQTT systems often have thousands of fine-grained topics, while Kafka works best with fewer, coarser topics. A common mapping writes many MQTT topics into one Kafka topic and keeps the original MQTT topic as the record key or a record header.

Does MQTT QoS 2 give exactly-once delivery into Kafka?

No. MQTT QoS applies to a single hop between client and broker. End-to-end from device to Kafka, most pipelines provide at-least-once delivery, so consumers should handle duplicates, for example with an idempotent write keyed by device and timestamp.

What is the easiest way to get MQTT data into Kafka?

Kafka Connect with an MQTT source connector needs no custom code and runs on existing Connect infrastructure. If your broker has a built-in Kafka bridge or rule engine, that is another option without extra services.

03ultramq/start

Talk to an engineer about your project.

Tell us what you want to connect, where it has to run and what the data should feed. We reply by email.

SUBultramq/#