How-to
Covering the QoS 0 loss window in Sparkplug MQTT transmission
Short answer
No. Sparkplug B requires NDATA and DDATA at QoS 0, so no supported MQTT Transmission setting publishes data messages at QoS 1 or adds broker acknowledgment. Enable the Rolling Buffer in the transmitter's History Store to replay messages lost in the transient TCP window, and design downstream consumers to accept duplicates, because replay is at-least-once.
On this page
No supported configuration publishes Sparkplug B data messages at QoS 1. The Sparkplug specification requires BIRTH, DATA and CMD messages at QoS 0, so MQTT Transmission offers no setting that raises NDATA or DDATA, and a QoS 0 publish carries no acknowledgment from the broker — a publish that completes locally does not prove the broker received it. The recovery is already on the transmitter: enable Rolling Buffer in the History Store so messages lost in the transient TCP window are retained and replayed after reconnection, then make everything downstream idempotent, because replay delivers at-least-once. This walkthrough shows where the window sits, how to close it, how to size it, and how to prove the recovery works.
Where the loss window actually sits #
Your path has three boundaries: MQTT Transmission on Ignition 8.3.3 publishes the Sparkplug B protobuf over TCP to HiveMQ; a Data Policy deserializes the protobuf into JSON inside the broker; the HiveMQ Enterprise Kafka Extension forwards that JSON to Azure Event Hubs. The unprotected window is the first boundary only.
A QoS 0 publish is sent once with no required acknowledgment — there is no PUBACK anywhere in the exchange. TCP catches most connection failures, but a message written to the socket just as the path fails can complete at Transmission while never arriving at the broker, and nothing at the publisher reports the difference. The History Store holds data produced while the connection is known to be down; the exposure is the transition. Records submitted just before Transmission detected the failure no longer count as pending, yet never landed, and that transition case is what the Rolling Buffer is designed to cover through replay.
Two more specification facts close off the obvious workarounds. Data messages are never flagged retained, so a consumer that subscribes late cannot pull them back from the broker. Command messages are QoS 0 as well, and a lost command is recovered through the rebirth sequence rather than retransmission — the diagnosis steps for Sparkplug rebirth command losses cover that case. For what the QoS levels themselves do and do not promise, read how offline message delivery compares with standard QoS before assuming a broker feature substitutes for an acknowledgment.
Enabling the Rolling Buffer #
- Open the MQTT Transmission configuration for the transmitter publishing to the broker.
- Confirm its History Store is configured and enabled. If data produced during a known disconnection is not being retained, correct that first — the Rolling Buffer builds on the store.
- Open the transmitter's History Store configuration and enable Rolling Buffer.
- Set the store's capacity and retention from the measured publication rate and the measured stored record size, not from a nominal tag count.
- Apply the configuration and check transmitter status for an active broker connection and an operational History Store.
- Leave DATA publication at the QoS the specification requires. Nothing here changes the message QoS, and nothing needs to.
Sizing the rolling window: 20 records per second across a five-minute outage #
The two inputs that matter are measured on your system; the values below are illustrative, in the units you will read them in.
- Peak publication rate: 20 records per second, taken from the transmitter's scan and deadband behavior at peak load.
- Recovery window: 300 s (five minutes), chosen to exceed the longest transient the link is expected to suffer.
- Average stored size: 1.5 kB per record, read from the History Store itself.
The rule in words: the store must hold the peak rate multiplied by the recovery window, in records, and that record count multiplied by the bytes per record, in bytes. Add engineering margin for bursts and payload-size variation.
The arithmetic:
6,000 records = 20 records/s × 300 s 9,000 kB = 6,000 records × 1.5 kB/record With a 2× margin: 12,000 records and 18 MB.
Decision: configure the History Store for at least 12,000 records and 18 MB, with retention past five minutes, and substitute your own measured rate and record size before committing the configuration. If your longest expected transient is shorter than five minutes, size to that interval instead — the window is the driver, not the rate alone.
Designing the consumer side for at-least-once #
Replay closes the loss window by accepting duplicates as the price. A record that did reach the broker can still sit inside the rolling window and publish again after reconnection, so the Data Policy transform and whatever reads Azure Event Hubs must apply each event once no matter how many copies arrive. Deduplicate on the stable event identity and source timestamp the payload already carries — never on the value, because two legitimate samples can hold the same value — and verify that key survives the protobuf-to-JSON transformation.
The same at-least-once discipline applies to store-and-forward backfill after a connection outage. It also governs consumers at the end of an MQTT to Kafka edge data pipeline without losing messages.
If any of these values feeds a safety function, treat this path as operational data only: QoS 0 delivery with replay is not a safety-rated channel.
Verifying recovery across the whole path #
- Record a baseline count, or an identifiable sequence of test changes, at four points: Transmission, broker ingress, transformed JSON output, and Azure Event Hubs.
- Break the broker network path while controlled changes keep generating, and note the exact disconnect time and connection status.
- Restore the path and watch the History Store drain; confirm the changes generated during the outage appear downstream.
- Create one controlled transient at the publish boundary — drop the TCP session mid-publish — and confirm the records retained by Rolling Buffer are replayed after reconnection.
- Compare the four observation points and classify any gap against the table below.
| Gap you observe | Where the loss sits | Correct response |
|---|---|---|
| Record missing at broker ingress | Publish boundary, transient window | Rolling Buffer replay; inspect the TCP path |
| Protobuf arrived, JSON absent | Data Policy processing | Correct the deserialization or transform |
| JSON present, Event Hubs record absent | Kafka Extension forwarding | Check extension delivery errors and destination acceptance |
| Duplicates only | Replay semantics | Idempotent consumer or deduplication |
Questions asked
Does adding a Sparkplug primary host or MQTT Engine close the loss window?
No. A primary host exchanges STATE and command traffic, and a subscriber such as MQTT Engine consumes the data, but neither gives the publisher an application-level acknowledgment of a DATA message. The missing proof is the MQTT PUBACK that a QoS 0 publication never generates, and no host-side component recreates it. Rolling Buffer replay remains the recovery mechanism in this architecture.
Are duplicate records after reconnect a fault?
No. Treat duplicate delivery as the normal outcome of recovery, not an error. The contract is at-least-once: each event may arrive one or more times and must be applied exactly once by the consumer. Alarm on gaps in the record sequence, not on duplicates, and keep deduplication at the consumer rather than trying to suppress replays at the transmitter.
Can the broker be configured to force QoS 1 on these topics?
A broker cannot deliver a message to subscribers at a higher QoS than the publisher used, and the acknowledgment the publisher receives is set by that publish. Since Transmission publishes NDATA and DDATA at the QoS the specification requires, there is nothing at the broker to upgrade into an acknowledgment. The recovery has to come from the publisher side, which is what the Rolling Buffer provides.
How do consumers resynchronize after the reconnect?
The transmitter republishes the birth messages when the connection is re-established, so consumers rebuild their metric metadata and aliases before data resumes. The Rolling Buffer then fills the data values that would otherwise be missing between the failure and the rebirth. Command messages are QoS 0 as well and are recovered through the rebirth mechanism rather than retransmission.
Should critical records get a separate QoS 1 path?
If specific records need per-message proof of delivery, yes. Publish those on a parallel topic at QoS 1 and treat that as a second path, not a modification of the Sparkplug one. The Sparkplug stream keeps births, metric metadata and distribution; the acknowledged path carries only the records whose receipt you must prove end to end.