IMD SolutionComputing LibraryShop main store

Troubleshooting

Historian stores zero after an MQTT edge node reconnects

Edge computing and IIoT Published 6 min read

Short answer

A 0 stored just before the true value at every MQTT edge reconnect is a reported pattern on Ignition 8.1.5 through 8.1.15 with the MQTT Transmission and Engine modules 4.0.6 through 4.0.10. Confirm it by querying stored records around each reconnect, exclude those zeros from process data, and remove them with a timestamped rule instead of a blanket zero filter.

On this page
  1. The signature in the stored records
  2. Confirm it with a query, not with the live tag
  3. Working the version span for modules 4.0.6 to 4.0.10 on Ignition 8.1.5 to 8.1.15
  4. Contain it while it persists
  5. Clean the stored zeros without losing real ones
  6. Questions asked

A 0 written to the historian just before the true value, every time an MQTT edge node reconnects, is a real stored record, not a display glitch, and it lands hardest on tags that did not change during the outage. Reports place it on Ignition 8.1.5 through 8.1.15 with the Cirrus Link MQTT Transmission and MQTT Engine modules 4.0.6 through 4.0.10, with nothing in the gateway log and no duplicate-entry quarantine to match. Confirm it by querying stored records around each reconnect, exclude those zeros from everything that consumes history, and remove them with a timestamped rule instead of a blanket zero filter. Upgrading inside the reported version spans has already been tried without effect, so put the effort into detection, flush ordering and cleanup.

The signature in the stored records #

The reports establish a specific set of facts:

  • A record with value 0 stores first at reconnect, and the actual value follows a few milliseconds later.
  • Tags that sat static through the outage show it most; tags that changed show it too.
  • It appeared on every module version the affected sites ran, and it grew more frequent over time without any module change.
  • It leaves no gateway log entry and does not line up with the occasional duplicate-entry quarantine.
  • Reference tags with working backfill still store the 0, so successful backfill does not rule the fault out.
  • Toggling 'Enable History Storage by Default' and adding the StoreAndForward custom property left the reported frequency unchanged, so repeating those changes is not a correction.

If your symptom is records missing entirely rather than a false value stored, that is a delivery fault instead, so start with fixing MQTT cloud broker data not reaching SCADA tags.

What fits the pattern, without being proven by it, is a tag that re-initializes to a default 0 while its quality is bad as the session returns, then takes its first real sample milliseconds later; a static tag has nothing newer arriving to collide with, which matches where the fault is most visible. The evidence does not isolate whether MQTT Transmission, MQTT Engine, reference-tag handling or the Ignition Tag Historian writes the zero, so treat it as a pipeline behavior to detect rather than a single-module bug to patch. The layout involved, Transmission on the edge nodes and Engine on the central gateway feeding a historian, is the same pipeline used when bridging OPC UA PLC data to MQTT with an edge PC.

Confirm it with a query, not with the live tag #

The live tag recovers immediately, so the fault stays invisible until you read stored records. Query the history provider's backing tables directly:

  1. Capture each reconnect timestamp from the edge node's connection status history or the gateway connection log. On cellular sites, where drops are frequent, collect these automatically.
  2. Query each affected tag's records from 1 second before to 5 seconds after every reconnect timestamp. The reported pair lands milliseconds apart, and the seconds after the timestamp also catch any store-and-forward flush that follows.
  3. Verify the pattern: a 0 immediately followed by the true value, which on a static tag equals the last value stored before the drop.
  4. Split the tags into static and changing groups; the static group is the stronger reproduction target.
  5. Record the Ignition release and MQTT module version on both sides, the Transmission nodes and the Engine gateway.
  6. Note whether each affected tag is a reference tag and whether it writes directly to the History provider.
  7. Correlate your timestamps with gateway logs and quarantine records, knowing the reported incidents matched neither, so silence there does not clear the fault.
  8. Force disconnect-and-reconnect cycles to reproduce on demand, including cycles where the source tags do not change.

Working the version span for modules 4.0.6 to 4.0.10 on Ignition 8.1.5 to 8.1.15 #

Inputs, each from the affected sites' own reports: the first module release showing the fault is 4.0.6 and the last is 4.0.10; one site moved from 4.0.8 to 4.0.10 mid-fault with no change; the platform range runs from Ignition 8.1.5 on the oldest Transmission node to 8.1.15 on an all-one-version site, with an Engine gateway at 8.1.14 in between.

Rule in words: the number of releases a report range covers is the last patch number minus the first, plus one, because both ends are included.

Arithmetic: for the modules, 10 minus 6 is 4 release steps, so 4 plus 1 is 5 releases, namely 4.0.6, 4.0.7, 4.0.8, 4.0.9 and 4.0.10. For the platform, 15 minus 5 is 10 release steps, so 10 plus 1 is 11 Ignition releases from 8.1.5 to 8.1.15. The one tested move, 4.0.8 to 4.0.10, is 2 steps inside that span and changed nothing.

Result: 5 consecutive module releases on 11 consecutive platform releases show the fault end to end, and no release inside either span is cleared by the reports.

Decision: any upgrade target inside 4.0.6 to 4.0.10 or 8.1.5 to 8.1.15 is a dead end, so skip it. The threshold for trialing a version change is landing outside both spans, which today means the 5.x module line on Ignition 8.3.x, accepted only after forced reconnect cycles come back clean. A later account at Engine 4.0.18 on Ignition Edge 8.1.28 describes a related zero-on-write cleared by an upgrade without naming the target release, so carry it as a lead for the vendor case, not as a prescription.

Contain it while it persists #

  • Set In-Order History on MQTT Transmission. Its documented behavior flushes all buffered historical data after a reconnect before live data resumes, which is exactly the ordering the fault's sequence points at, and the store-and-forward caveats in the MQTT History documentation demand chronological flush order anyway. Verify it with forced cycles after enabling.
  • Exclude the reconnect zeros from every consumer, including trends, reports and anything computed from history, using the timestamped rule below rather than a global value filter.
  • Leave backfill running; it works through this fault, and the same chronological-order requirement applies when you backfill trend history from a plant historian.
  • Keep the edge nodes' clocks synchronized, or the reconnect timestamps you query around will not line up with the stored records.
  • Open the vendor case carrying the query output: reconnect timestamps, the stored record pairs, versions on both gateways, and the reference-tag status of each affected tag.

Clean the stored zeros without losing real ones #

Delete or flag a stored record only when three conditions hold together: its value is 0, its timestamp sits inside a reconnect window, and the record immediately after restores the value the tag held before the drop. A blanket rule that discards zeros will eat legitimate process zeros, and the reports give you no way to tell them apart once deleted. Export the affected records before deleting anything; if this history feeds regulated reporting or alarm reviews, that export is your audit trail.

Acceptance after cleanup or any configuration change: force reconnect cycles that include static tags, and pass only if no unexplained zero precedes the restored value while legitimate zeros and backfilled records survive untouched.

Questions asked

Is the reconnect zero stored by MQTT Transmission or by the Tag Historian?

The reports confirm the stored record, not the component that writes it. It shows with MQTT Transmission and Engine 4.0.6 through 4.0.10 on Ignition 8.1.5 through 8.1.15, in tags writing directly to the History provider and in reference-tag setups with working backfill, so neither side is cleared by the evidence. Detect and filter it at the historian whichever side you later patch.

Will upgrading the MQTT modules inside 4.0.6 to 4.0.10 stop it?

No. The fault spans five consecutive module releases, 4.0.6 through 4.0.10, across eleven Ignition point releases from 8.1.5 to 8.1.15, and one site that moved from 4.0.8 to 4.0.10 saw no change. Trial a version change only outside both spans, today the 5.x modules on Ignition 8.3.x, and accept it only after forced reconnect cycles come back clean.

Which stored zeros can I delete?

Only zeros that meet all three conditions at once: value 0, timestamp inside a reconnect window, and a following record that restores the pre-drop value. Where you cannot establish that the process cannot legitimately sit at zero, flag the record instead of deleting it. Export the affected records before you delete anything.

Should I disable store-and-forward until the fault is fixed?

No. Backfill keeps working through the fault, with sites reporting reference-tag backfill succeeding while the 0 is still stored, so disabling it converts a removable false record into a permanent data gap on every drop. Keep store-and-forward enabled, enforce chronological flush order, and let the cleanup rule remove the zeros.

Keep reading

  • Troubleshooting

    Troubleshoot missing data in a Node-RED InfluxDB Grafana stack

    Work the chain in order: acquisition into Node-RED, the InfluxDB service and its storage mount, the write settings, then the Grafana query. In a 2.x stack most empty panels trace to a wrong token, bucket, URL or a controller clock that stamps points outside the panel's time range.

    Edge computing and IIoT

  • Explainer

    MQTT to Kafka edge data pipeline without losing messages

    Keep MQTT on the sensor side of the plant, put the Kafka-class durable log on the site or cloud side of the uplink, and give the boundary one disk-backed store-and-forward buffer sized to the longest outage you will ride through. At 500 messages per second and an 8-hour outage, that queue holds 14.4 million messages and about 9.6 GB — set the ceiling above it and alarm on overflow.

    Edge computing and IIoT

  • How-to

    Route MQTT sensor data into Grafana on an edge PC

    An MQTT broker forwards each message once and keeps no history, so Grafana cannot trend from it directly. Subscribe with Node-RED, write each payload into InfluxDB, set a 365-day retention policy, and point a Grafana data source at that database. For the example below — 720 signals sampling every 5 s — that is about 0.9 TB for a year; provision roughly 2 TB.

    Edge computing and IIoT