Troubleshooting
Getting backfilled Sparkplug history through Ignition reference tags
Short answer
Your settings are right; order is the missing condition. Backfill reaches the reference tags only when Store Historical Events is OFF so backfilled samples drive the Engine tag, and only when the edge replays its backlog oldest-to-newest past metric timestamp validation. Out of order, disable metric timestamp validation and keep the replayed value out of live control reads.
On this page
Your three settings are correct, and reference tags are not the limitation. With Store Historical Events OFF, a backfilled Sparkplug sample becomes an ordinary value change on the Engine tag, and reference tags propagate value changes, timestamps included, into their own history. The condition that breaks it is order: metric timestamp validation in MQTT Engine refuses any metric stamped older than the last accepted one, so a backlog that replays out of order leaves the reference tags silent. Test the replay order during one controlled reconnect. In order, the settings you already have pass backfill through. Out of order, disable metric timestamp validation and keep the replayed value out of anything that needs the current process state.
Where the backfilled sample stops #
A backfilled sample travels edge node → broker → MQTT Engine → Engine tag → reference tag → reference-tag history. Find the first boundary it fails to cross; everything upstream of that boundary is proven and needs no changes.
| Observation | What it proves | Next check |
|---|---|---|
| No backfilled records anywhere on the Engine tag | Nothing downstream is proven | Link state, broker connection, subscription, and whether the edge publishes its stored samples after reconnecting |
| Engine tag history holds the backfill with original timestamps, but the tag value never stepped through the backlog | Transport, decode and Engine ingestion all work | Store Historical Events, then metric timestamp validation |
| Engine tag steps through the backlog, reference-tag history stays empty | The events reach the live tag path | Reference member history enabled, Allow Back-fill Data on the owning provider |
| Real-time changes historize through the references, backfill does not | The reference path and live history work | Replay order, which is the deciding measurement |
When nothing reaches the Engine tag at all, work the upstream buffer first: how MQTT store and forward backfills after a connection outage covers that stage. When the Engine tag already holds the missing samples, leave the broker address, port and subscription alone; the samples crossed that path.
Why a stored record never reaches a reference tag #
A reference tag reacts to value changes on its source tag. It does not read the source tag's stored history, and putting it inside a UDT adds organization, not a subscription to historical events. With Store Historical Events ON, MQTT Engine writes historical events straight into tag history without driving the tag: history fills in, the value never moves, and the reference tag has nothing to propagate. That split — history correct on the Engine tag, nothing on the references — is the signature of this path.
With the setting OFF, a backfilled sample arrives as a tag change carrying the metric's own timestamp. The reference tag sees a source change and re-evaluates, and with Allow Back-fill Data ON the provider that owns the references accepts a record stamped earlier than its newest stored value. Your live path already proves the references and their history settings work; only the historical path differs, and it differs in one place: whether the change ever reaches the tag.
If Store Historical Events is confirmed OFF and the references still stay silent through a reconnect, the samples are being refused before they can drive the tag. That refusal is metric timestamp validation dropping an out-of-order backlog, and the order test below decides it.
The four conditions that must hold together #
- Store Historical Events OFF in the MQTT Engine configuration, so historical Sparkplug data is written to the Engine tag and can trigger the reference path.
- Allow Back-fill Data ON on the tag provider that owns the reference tags, so records stamped older than the newest stored value are admitted.
- History enabled on the reference tag members that must record the backfill.
- A backlog that replays oldest-to-newest, or metric timestamp validation disabled.
Three of these are already set in your case. The fourth decides it: when the edge replays out of order, validation drops each older-than-last metric before it can drive the tag, and the reference tags see nothing regardless of the other three settings.
Testing replay order after a 20-minute outage at 10-second logging #
The order test needs one controlled reconnect and three timestamp readings. The values here are an illustrative test case; substitute your own outage window and logging rate, read from the connection log and the edge node's history configuration.
Inputs:
- Outage window: link drops at 09:40:00, reconnects at 10:00:00, which is 20 min.
- Logging rate: one Sparkplug sample per metric every 10 s.
- Stored interval: 20 min × 60 s/min = 1200 s of samples.
The rule in words: an in-order replay carries metric timestamps that only advance. The first backfilled timestamp should equal the outage start, and the last should be the sample logged just before the reconnect.
Arithmetic:
- Samples in the backlog: 1200 s ÷ 10 s per sample = 120 samples.
- Expected first timestamp: 09:40:00.
- Expected last timestamp: 09:40:00 + (120 − 1) × 10 s = 09:40:00 + 1190 s = 09:59:50.
- Mid-backfill check at message 61: 09:40:00 + 60 × 10 s = 09:50:00.
Run the reconnect with metric timestamp validation temporarily disabled and watch the Engine tag's timestamp as the backlog replays. If message 1 reads 09:40:00, message 61 reads 09:50:00 and message 120 reads 09:59:50, with no backward step anywhere between, the backlog is in order: re-enable validation, repeat the reconnect once, and confirm the reference-tag history fills with the original timestamps. One single backward step is enough to take the next section's branch instead.
Taking the out-of-order branch #
When the backlog replays out of order, leave the three settings in place and disable metric timestamp validation in the MQTT Engine configuration. Every metric, live and historical, then drives the Engine tag; the references follow; the owning provider's backfill setting keeps the original timestamps in storage. Nothing here corrupts the historian, because records store under their own timestamps rather than arrival order.
The cost runs on the display and in logic: while the backlog replays, the Engine tag and the reference tags show an older historical value rather than the current one, until the replay finishes and the live value takes over. Keep replayed values out of any control or interlock decision that needs the present process state.
Fixing the order at the source is worth one look before settling for disabled validation: if the edge publishes through MQTT Transmission, its transmitter History Settings carry an In-Order History configuration. On the publisher side generally, configuring MQTT store-and-forward for data backfill covers how the edge stages and replays its backlog. If the central side later shows the outage as one long held value instead of a gap, that is a separate display effect, covered in why Edge historian gaps look like held values centrally.
Applying the fix in order #
- Confirm the Engine tag's history holds the backfilled samples with their original timestamps. If it does not, repair the upstream path first; nothing downstream recovers samples that never arrived.
- Set Store Historical Events OFF in the MQTT Engine configuration.
- Set Allow Back-fill Data ON on the tag provider that owns the reference tags.
- Keep history enabled on each reference tag member that must record the backfill.
- Run the controlled reconnect of the order test with metric timestamp validation temporarily disabled, and record the Engine tag timestamps.
- Timestamps only advance: re-enable metric timestamp validation, repeat the reconnect once, and confirm the reference-tag history fills.
- Any backward step: leave metric timestamp validation disabled for that edge node, and keep the replayed value out of current-state logic.
Steps 2 through 4 are what you already have. Step 5 is the measurement your case is missing, and steps 6 and 7 are its two outcomes.
Questions asked
Is this a limitation of reference tags or UDTs in Ignition?
No. A reference tag propagates value changes from its source tag; it never reads the source's stored history, and a UDT adds organization, not a historical subscription. Once backfill drives the Engine tag as value changes, the reference members historize those changes with the metric timestamps. No redesign of the UDT is needed.
What does metric timestamp validation actually protect?
It keeps a tag from stepping backward in event time when a stale or replayed message arrives. Disabling it does not corrupt the historian, because records store under their own timestamps rather than arrival order; the exposure is on the live side, where the displayed value can sit at an older sample until the replay finishes.
Do the original sample timestamps survive into the reference-tag history?
Yes. Each propagated change carries the metric's own timestamp, and Allow Back-fill Data ON on the owning provider is what admits records stamped earlier than the newest stored value. Without that provider setting, the tag still updates but the historian refuses the older rows.
How do I force the controlled reconnect the order test needs?
Break the path the subscription actually uses for a known window: stop the broker connection or disconnect the edge's network route, then restore it and watch the Engine tag timestamps as the backlog replays. A window of about 20 minutes at your normal logging rate keeps the backlog readable by hand.
Does turning Store Historical Events off stop the Engine tag from storing history?
No. It changes the path, not the record: historical samples arrive as tag value changes and historize from those changes exactly like live data, each with its own metric timestamp. What you give up is the direct historian-insert path that filled history without ever moving the tag.