Explainer
Does an MQTT broker store historic data you can query
Short answer
No. An MQTT broker transports publications and keeps only delivery state: the newest retained value per topic and messages queued for offline persistent sessions, with no time-range query and nothing else to query. To move many timestamped samples in one delivery, the publisher must buffer them and send a multi-sample payload such as Sparkplug B; to query history later, something must store it when it arrives, in a historian or database.
On this page
- What an MQTT broker actually keeps
- Why retained messages and wildcards cannot backfill
- What one timestamp in a JSON payload means
- How Sparkplug B carries buffered samples
- The two designs that keep history
- What a subscriber recovers after a 1 hour outage at 1 sample per second
- The decision for each case
- Questions asked
No. An MQTT broker is a router, not an archive. It moves each publication from publishers to the subscribers whose subscription matches the topic, and the only things it keeps are delivery state: the newest retained publication on each topic and messages queued for one offline client with a persistent session. Once a message has been delivered, the broker's copy is gone, and there is no interface that returns everything the broker saw between two times. A JSON array of timestamped samples therefore cannot be pulled out of a broker, no matter how it is formatted. To move many timestamped samples in one delivery, the publisher must hold them and send them together in a payload built for that, which Sparkplug B can do. To query history afterwards, something must store it as it arrives: a historian or a database.
What an MQTT broker actually keeps #
A broker runs two stores, and both exist for delivery, not for history.
A retained publication is stored one per topic and handed to anyone who subscribes later: it is the broker's last-known-value service. A new retained publication on the same topic replaces the old one, and a publisher clears the slot by sending an empty retained payload. The protocol carries no publication timestamp, so the retained value also arrives without a sample time.
A persistent session is stored per client ID and serves exactly one client. While that client is offline, the broker holds the messages its session settings say to keep and delivers them on reconnect; each entry leaves the queue once it is delivered. The handshake state behind quality-of-service delivery is bookkeeping and disappears when the delivery completes.
| Store on the broker | What it is for | What it cannot do |
|---|---|---|
| Retained publication, one per topic | Give every later subscriber the last published value | Keep earlier publications or their sample times |
| Persistent session queue, per client ID | Hold messages for one offline client until it reconnects | Redeliver to anyone else, or answer a time-range query |
| In-flight quality-of-service state | Track delivery handshakes | Outlive the completed delivery as retrievable history |
Nothing in that table can be queried by time range. A broker offers publish and subscribe; it does not offer select.
Why retained messages and wildcards cannot backfill #
A wildcard subscription returns matching publications from the moment it is active, plus one retained value per matching topic. Widening the filter widens the live stream, not the past: publications already delivered and discarded are unreachable, from that client or any other.
The missing timestamp compounds it. A consumer that reconnects and receives one value per topic cannot place those values on a trend, because nothing in the message says when they were published. And when an edge publisher reconnects after a gap and the receiving system still shows missing points in that window — a commonly reported symptom with store-and-forward setups — the usual cause is that the samples in the gap never reached any store. Replay can only send what the publisher kept.
None of that is a broker defect, and no broker setting repairs it; it is what publish/subscribe transport is defined to do. What does need fixing first is different: if publications are not arriving at all, that is a transport fault, and troubleshooting an MQTT publisher that cannot reach its broker is where that work starts.
What one timestamp in a JSON payload means #
A payload carrying one message-level timestamp with several values is a snapshot: every value in it belongs to the same instant. That is the reported behavior of the format the question calls JSON schema A — one timestamp, multiple values — and it is a sensible design for compact current-state messages. A JSON array is valid syntax, but syntax is not semantics: an array gives each element its own acquisition time only if the receiving driver's documented schema defines per-element timestamps and stores them.
So check one thing in your driver's documentation before designing around it: whether the timestamp covers the whole message, a group of values, or each sample. If the schema defines a single message-level timestamp, adding an array or extra time fields changes nothing — the parser reads the one timestamp it knows and ignores or rejects the rest. No formatting trick turns a snapshot format into a time series; that takes a format defined for per-sample times at both ends.
How Sparkplug B carries buffered samples #
Sparkplug B supports multiple time and value entries in the same payload, so a publisher that buffers samples during a gap can deliver the whole block in one message. The payload is a structured binary format rather than free-form JSON, and to the broker it is just bytes: any broker that accepts the message size moves it unchanged. The capability lives at the endpoints.
Two independent checks decide whether it works. The publisher must be able to read its buffer and encode each sample with its original acquisition timestamp — an encoder that stamps the whole message at send time has already thrown the history away. The receiving driver must interpret those per-sample timestamps as measurement times and write them as history: a receiver that only updates a current-value display will consume the batch and keep nothing queryable. Metric names or aliases and data types must match across the two ends, and each batch must fit the most restrictive size limit among publisher, broker and subscriber decoder.
One publication is also one transport message, not an all-or-nothing database transaction. If the receiver must commit a batch atomically, its documented commit and error behavior decides that — not the protocol. And if you are still choosing the controller-to-SCADA route, the trade-offs between OPC UA and MQTT Sparkplug sit one decision upstream of this.
The two designs that keep history #
History survives an outage or answers a query only if some component with storage holds the samples. Two designs do it, and they combine.
Design 1: buffer at the publisher, send multi-sample payloads #
The publishing device or gateway holds samples in its own memory or local store while the link is down, then sends each block as one multi-sample payload. That retention is the publisher's job, and it is the same discipline any batching or averaging function needs: an application that computes over samples must retain them in an array of its own first — why a mean needs an array that retains the samples. Downstream, the receiving application writes every sample at its own timestamp into real storage; writing timestamped history back into a SCADA server is exactly that receiving side.
Design 2: store at arrival, query the store #
Live publications are written into a historian or database as they arrive, and every later question — trends, reports, gap analysis — goes to that system, never to the broker. Once the history is in the store you can calculate over it, such as a historic delta from sparse samples, and you can expose historian latest values over OPC UA or MQTT for consumers that still want MQTT-style current values.
Anything with a retention obligation behind it — batch records, environmental data, safety-related events — needs design 2, or a real store behind design 1: a transport is not a record.
What a subscriber recovers after a 1 hour outage at 1 sample per second #
The case: a gateway publishes one machine-health topic at 1 sample per second with the retain flag set, its uplink drops for 1 hour, and a consumer connects fresh after recovery with no session from before the gap.
Inputs, and where each comes from:
- Sample rate: 1 sample per second — an assumed typical rate for a current-value topic, an illustrative input, not a measured one.
- Gap length: 1 hour, which is 3600 s — an assumed outage length.
- Retain flag set on every publication — the documented mechanism by which a broker keeps a last value for future subscribers.
- Broker behavior: exactly one retained publication is kept per topic, and a new retained publication replaces the old one — documented retained-message behavior.
- Consumer session: none from before the gap, so no queue exists for this consumer.
The rule in words: during the gap the publisher produces one sample per second, and every retained publication overwrites the single retained slot for its topic. On reconnect, the consumer receives what the broker still holds — the newest value on each retained topic — with no sample time attached.
Arithmetic:
- Samples generated during the gap: 1 sample/s × 3600 s = 3600 samples.
- Samples the broker still holds for that topic: 1.
- Recovered fraction: 1 ÷ 3600 = 0.00028, about 0.03% of the gap.
- Missing samples: 3600 − 1 = 3599.
Result: the consumer gets 1 of the 3600 samples, roughly 0.03% of the gap, undated.
Decision: if the requirement is every sample — 100% coverage of the gap, which quality and compliance calculations usually demand — the retained topic fails by 3599 samples, and no broker setting changes that number. The gap has to be covered upstream: the gateway buffers and sends the block as multi-sample payloads, or a historian has been receiving the live stream all along. If the requirement is only the current value on connect, the retained topic is the right tool and already does the job.
The decision for each case #
- Current value on connect is all you need: retained publications. Nothing else to build.
- Every sample must survive an outage: buffer at the publisher and send multi-sample payloads such as Sparkplug B, into a receiver that archives per-sample timestamps. Verify both ends — protocol capacity and receiver behavior are separate checks.
- You need to query trends later: store at arrival in a historian or database and query that system. If the open question is whether you need a full historian or just a tag server, do you need a historian or just an OPC tag server lays out that choice.
- The broker's role is the same in every case: move publications and keep delivery state. History lives in a publisher that buffers or a store that archives — never in the broker.
Questions asked
Does QoS 1 or QoS 2 make the broker keep messages as history?
No. QoS levels govern delivery confirmation between the publisher, the broker and each subscriber. Queued QoS 1 and 2 messages exist only to reach a specific offline client whose session the broker keeps, and each queued message leaves the broker once it has been delivered. Nothing in quality of service stores a copy for later retrieval or records a sample time you can query.
How long does a retained message stay on the broker?
Until the next retained publication on that topic replaces it, or until a publisher clears it with an empty retained payload. It is not an archive with a retention period: there is exactly one per topic at any moment, the protocol records nothing about when it was published, and whether it survives a broker restart depends on the broker's own persistence configuration.
Can Sparkplug B carry buffered samples through any MQTT broker?
Yes, as far as the broker is concerned. A Sparkplug B payload is just bytes to the broker, so any broker that accepts the message size moves it unchanged. The capability lives at the endpoints: the publisher must assemble the samples with their own acquisition times, and the receiving driver must interpret those times and store the samples as history.
Does a persistent session give a reconnecting client the full gap?
Only what was queued for that one client while it was away, at the quality-of-service levels the broker keeps for its session. Messages published before the session existed, publications another client already consumed, and anything discarded under the broker's queue limits never come back. A queue is a delivery buffer for one client ID, not a queryable archive.