Explainer
MQTT to Kafka edge data pipeline without losing messages
Short answer
Keep MQTT on the sensor side of the plant, put the Kafka-class durable log on the site or cloud side of the uplink, and give the boundary one disk-backed store-and-forward buffer sized to the longest outage you will ride through. At 500 messages per second and an 8-hour outage, that queue holds 14.4 million messages and about 9.6 GB — set the ceiling above it and alarm on overflow.
On this page
MQTT and Kafka belong on opposite sides of the plant uplink, and the uplink is where edge messages die. Keep MQTT on the sensor side of the network, put the Kafka-class durable log on the site or cloud side, and place exactly one disk-backed store-and-forward buffer at the boundary, sized to the longest outage you intend to ride through. That placement removes the two structural causes of loss — a broker too heavy for its host, and a buffer that was never sized or never bounded — and leaves the session settings, which the table below decides. The sections walk the chain hop by hop, give the buffer arithmetic, list the settings that determine delivery, and end with the outage test that proves the design.
Where MQTT ends and Kafka starts #
Trace one reading from sensor to analytics and the pipeline has four hops, each with its own failure mode and its own response:
| Hop | Carrier | Failure mode | Design response |
|---|---|---|---|
| Sensor to gateway | Fieldbus, serial or local IP | device reboots, polling gaps | timestamp assigned at the gateway |
| Gateway to site broker | MQTT over the plant LAN | broker memory exhaustion | broker sized to the hardware, queues bounded |
| Site to central log | MQTT bridge or MQTT-to-Kafka producer across the uplink | uplink loss, latency | the single disk-backed store-and-forward buffer |
| Central log to analytics | Kafka consumers | consumer lag, partition skew | lag monitored, retention sized to outage length |
The rule the table encodes: MQTT, which carries per-client session state, stays left of the uplink; the Kafka-class durable log sits right of it; and one buffer owns the crossing. A common carrier for that hop is a small consumer service such as a Telegraf MQTT consumer producing to Kafka, and when it fails to start on a broker-resolution error the cause is configuration rather than the uplink; resolving the brokers error in a Telegraf Kafka consumer documents that specific fault. Protocol alternatives change the carrier names, not the rule: a NATS server with its persistence layer or a managed edge runtime crosses the same boundary through the same buffer.
If the left end of your chain is PLC tag data rather than plain sensor messages, bridging OPC UA PLC data to MQTT with an edge PC covers the hop that turns it into MQTT topics.
Decision: draw your own deployment against the table. Every hop that crosses the uplink sends its messages through the one buffer, and no other process writes to the WAN.
Why the Kafka-class log does not run on the gateway #
A Kafka-class broker leans on two memory consumers that grow with partitions and retained data: its heap and the operating system's page cache, which the design deliberately uses as its write and read buffer. Past a handful of topics, that appetite is measured in gigabytes — server-class hardware, not a fanless box with a few gigabytes of RAM. Kafka-API-compatible brokers such as Redpanda remove the JVM but not the server-class expectations, so read the vendor's published minimum host requirements instead of assuming a fit. A lightweight MQTT broker carries a far smaller footprint because it holds per-client state and a bounded queue, which is exactly what the gateway hop needs. Choosing between a log-style broker and a traditional message queue is a separate question, and a practical decision framework for Kafka versus RabbitMQ works through it — but neither class belongs on the constrained gateway.
Measure rather than guess whether a candidate fits the gateway:
- Install the candidate broker on the actual gateway image, not a laptop approximation.
- Replay recorded sensor traffic at the expected peak rate, then again at two to three times that rate.
- Log resident memory, CPU load and storage write rate over a full day; memory that keeps climbing after warm-up signals unbounded queues or per-client state.
- Set a hard queue or retention limit so the broker sheds load by policy instead of being killed by the operating system's out-of-memory handler.
If the replay fails — on a small gateway it usually does — the fix is placement, not tuning: the log moves to a server-class host on the site or cloud side, and the gateway keeps the small MQTT broker plus the buffer. Where the site side needs its own computer to host that broker and the bridge, a fanless industrial mini PC is the host class built for 24/7 cabinet duty. Below the scale where any log broker earns its footprint — one line, a handful of assets — a self-hosted Node-RED, InfluxDB and Grafana stack does the same job, and sizing a self-hosted Node-RED InfluxDB Grafana historian works the numbers for that case.
Decision: the Kafka-class log lives on server-class hardware right of the uplink; the gateway keeps the MQTT broker and the buffer and nothing heavier.
Size the one buffer to the longest outage #
The boundary buffer is the bridge's or producer's disk-backed store-and-forward queue. Mainstream automation platforms ship one as a standard feature of their MQTT modules, and on reconnect it drains the backlog to the central side in queued order. Exactly one such queue exists in this design: a second one inside another application doubles the disk load and hides the overflow point.
Sizing takes four inputs: peak publish rate, the longest outage you choose to ride through, payload size, and queue overhead. Worked example for one line:
- Rate: 500 messages per second, from 500 tags publishing at 1 Hz, one MQTT message per tag per cycle.
- Outage target: 8 hours, one shift, which is 28,800 s.
- Buffered messages: 500 msg/s × 28,800 s = 14,400,000 messages.
- Bytes per message: 512 B average payload plus roughly 30% for topic names, sequence numbers and queue metadata: 512 B × 1.3 = 665.6 B.
- Queue volume: 14,400,000 × 665.6 B = 9,584,640,000 B ≈ 9.6 GB.
Decision: set the queue ceiling at 16 GB, headroom above the 9.6 GB one-shift figure, and alarm on overflow. A 60-hour weekend, Friday 18:00 to Monday 06:00, is 216,000 s and at the same rate needs 500 × 216,000 × 665.6 B = 71,884,800,000 B ≈ 72 GB — a different hardware decision: dedicate a larger partition to the queue, or cut the rate with edge aggregation. Pick the ride-through deliberately, because a bounded queue drops by policy once it fills, and the drop count must surface as a metric, never as silence.
Wear depends on when the queue writes. If the bridge persists every message to disk even while the uplink is healthy, the steady write load is 500 msg/s × 665.6 B × 86,400 s ≈ 28.7 GB per day, and the drive's rated total endurance has to absorb that for years; if it writes only during outages, wear tracks outage time and is negligible. Find which behavior your bridge has before you select the storage.
One safety note: nothing on the analytics side of this pipeline drives a machine safety function. Store-and-forward bounds data loss, not response time, so safety logic stays in the machine's own controller.
Session settings that decide delivery #
Most outage losses trace to session configuration, not to the protocol. Set these on every publisher and on the bridge:
| Setting | What it does | Set it to |
|---|---|---|
| QoS 0 | no acknowledgement; lost on any disconnect | replaceable, high-rate telemetry only |
| QoS 1 | at-least-once; duplicates possible | default for measurements; consumers tolerate duplicates |
| QoS 2 | exactly-once; a four-packet handshake, double the round trips of QoS 1 | weak links only, used sparingly |
| Persistent session | broker queues QoS 1 and 2 traffic for an offline subscriber | enabled on the bridge; bound the session queue |
| Keep-alive interval | how fast a dead TCP link is detected; the broker disconnects after 1.5 times the interval with no traffic | matched to the uplink's real flap behavior |
| Last Will and Testament | broker publishes a status message on an ungraceful disconnect | configured, so the asset model can flag a gateway offline |
| Client ID | identifies the session to the broker | unique and stable per gateway |
Duplicate client IDs are the classic hidden fault: two gateways sharing one ID evict each other in a reconnect loop that looks exactly like network instability and is a configuration error.
Decision: QoS 1 on measurement topics, persistent session with a bounded queue, unique client IDs, keep-alive matched to the link, Last Will configured.
Timestamps assigned at the gateway #
Store-and-forward replay is chronological; the queue drains in the order messages arrived. That is the feature and also the trap: a timestamp assigned after the uplink, at cloud ingest, encodes arrival order, so every reading buffered during the sizing example's outage would arrive wearing the restore instant as its event time. Assign the timestamp at the gateway, next to the sensor, and the backlog replays into history in true event order.
The replayed backlog reaches the historian as backfill; backfilling PI from an existing plant historian covers the historian side of that arrival. On the log side, map each MQTT topic to a partition key built from the asset ID so per-asset order survives parallel partitions and consumers.
Decision: gateway-assigned timestamps, asset ID as the partition key, chronological backfill that consumers accept without reordering.
The outage test that proves the design #
Run this once before rollout and again after every change to broker, bridge or firmware:
- Inject a numbered series from a simulated sensor at the gateway, each message timestamped at the gateway.
- Cut the uplink for longer than the typical outage while injection continues, and confirm the Last Will offline status appears on the central side.
- Restore the link and confirm every sequence number arrives: no holes, duplicates counted and handled.
- Confirm gateway memory and queue depth stayed inside the limits measured in the capacity replay.
- Reboot the gateway mid-outage and repeat; only a disk-backed queue passes this step, which is the point of it.
- Stop the central consumer for a while, restart it, and confirm consumer lag drains to zero without the site broker's memory climbing.
Decision: the pipeline is production-ready when the numbered series returns complete after both an uplink cut and a mid-outage power cycle.
Questions asked
Does QoS 1 protect messages across a gateway reboot?
No. QoS 1 covers one client-to-broker hop only, so it protects the LAN hop while both ends are running. If the publisher's outgoing queue lives in RAM, a power cut empties it before QoS matters. Surviving a reboot needs the queue on disk plus a persistent session, and the only proof is the mid-outage reboot test: cut power during a planned outage and confirm the numbered series still arrives complete.
What should the buffer drop when an outage outlasts it?
Choose a policy in the bridge configuration before it ever happens. Oldest-first drop keeps the newest state, which suits condition monitoring where the current reading matters most; newest-first keeps history when trend continuity matters more. Whichever you pick, count the overflow and expose it as a metric, because an overflow nobody can see is a silent gap in every model built on the data.
Can the Kafka log run inside the plant instead of the cloud?
Yes. The rule is a boundary, not a destination: the Kafka-class log may run on a virtualized server in the plant's server room as long as it sits on the site side of the constrained uplink and gets server-class memory and disk. What must never happen is co-locating it on the field gateway, where its page-cache appetite competes with the MQTT broker and the buffer.
May the MQTT broker and the Kafka bridge share one gateway?
Yes, and on a small site they usually do. Run them as separate processes with the broker bound to the LAN interface and the bridge as the only process that owns the WAN connection. Set a memory ceiling on each and restart them independently, so a bridge crash cannot take sensor-side delivery down with it and the other way around.
How fast does the buffer drain after a long outage?
At the uplink's sustained bandwidth, not at the sensor rate. The 9.6 GB backlog from the sizing example is 76.8 gigabits; over a 10 Mbit/s link that is 7,680 seconds, about 2.1 hours of pure transfer. Expect consumer lag during the drain and size the historian's lag tolerance to the same outage number rather than to steady-state traffic.