IMD SolutionComputing LibraryShop main store

Troubleshooting

Remote tag history stops when gateway redundancy fails

Edge computing and IIoT Published 6 min read

Short answer

A running Gateway Network connection proves the transport, not the services behind it. In the reported case the redundant pair hosting the history providers had detected thread deadlocks on the backup and a disconnected redundancy peer on the primary after a direct upgrade to 8.3.4, so gateway and redundancy health must be restored before any remote provider is reconfigured.

On this page
  1. What a running connection proves
  2. Two failures that look like one
  3. The diagnostic path
  4. Fault count on the reported pair after the 8.1.32 to 8.3.4 jump
  5. Capture the deadlock before you clear it
  6. Restore the pair, then the connection bookkeeping
  7. Reconnect the remote history provider last
  8. Questions asked

A remote tag history provider showing ERROR / Gateway not connected while every Gateway Network connection reports a running status is telling you the transport is healthy and the history-hosting gateway is not answering service requests. In the reported case the redundant pair hosting the history providers was itself faulty after a direct upgrade from Ignition 8.1.32 (b2023091211) to 8.3.4 (b2026030309): the backup carried a performance warning that thread deadlocks had been detected, and the primary reported its redundancy peer as not connected. Restore gateway and redundancy health first. Reconfiguring the remote provider before that only chases the symptom, because a sick host makes every mapping look broken.

What a running connection proves #

A Gateway Network connection in a running state proves two gateways can exchange traffic. It does not prove the far gateway can process a provider-discovery request or a history query. A deadlocked gateway keeps its network threads alive while the subsystems that answer those requests are stuck, which is exactly how you get a running connection sitting next to ERROR / Gateway not connected.

The request path has four stages, and each fails differently:

  • the configuration view on the requesting gateway;
  • the Gateway Network connection to the history host;
  • the services on the history-hosting gateway, including its redundancy state;
  • the history provider itself on that gateway.

A remote history provider is a link, carried over the Gateway Network, to a history provider that lives on another gateway. The reported symptom clears the second stage, since every machine shows running, and puts the fault in the third: the hosting gateway cannot enumerate its own providers, so the requesting gateway has nothing to fetch.

Two failures that look like one #

The report contains two distinct problems, and they need different recovery work:

  • History that was never written. If the tables you expect are absent on the host, that is a logging problem on the host; see why Ignition tag historian tables are not created before touching anything on the remote side.
  • History that exists but cannot be served. That is this fault: a service-responsiveness problem on the hosting pair.

The query mechanics are separate from the transport and are covered in querying tag history from a remote tag provider. Keep the two investigations apart or you will fix the retrieval path while the store stays empty.

The diagnostic path #

Work this order and write down what each step returns:

  1. On the requesting gateway, record the exact provider status string and the time it appeared.
  2. On every gateway, list each Gateway Network connection with its identity, direction and state, and mark every incoming connection that has no matching outgoing connection on the other machine.
  3. On both members of the hosting pair, read the redundancy status: each member's view of its peer, and which member is active.
  4. Read the performance warnings on both members. A warning that thread deadlocks have been detected stops services; it is not noise.
  5. Capture logs and thread diagnostics from the deadlocked member before anything restarts it.
  6. In a fresh configuration view, select the intended history host and watch the provider list. An empty list, or one carried over from a previously selected gateway, is a failed discovery, not a result.
  7. Compare several requesting gateways. Failures that converge on the same host move the investigation to that host.

Fault count on the reported pair after the 8.1.32 to 8.3.4 jump #

Inputs, each from the reported case:

  • Hosting: one redundant pair, 1 primary and 1 backup.
  • Upgrade: 8.1.32 (b2023091211) direct to 8.3.4 (b2026030309), 0 intermediary upgrades.
  • Backup: 1 performance warning, thread deadlocks detected.
  • Primary: redundancy peer reported not connected.
  • Primary: 1 incoming Gateway Network connection from the backup, running.
  • Backup: 0 corresponding outgoing connections.

Rule in words: a hosting pair can serve provider discovery only when three counts are zero — deadlock warnings on either member, members reporting their peer disconnected, and incoming connections with no matching outgoing connection on the peer.

The arithmetic:

  • Members to check: 1 primary + 1 backup = 2.
  • Deadlock warnings: 1.
  • Peer-disconnect reports: 1.
  • Unpaired connections: 1 incoming − 1 outgoing = 1.
  • Total health faults: 1 + 1 + 1 = 3.

Result: 3 faults on the pair that must answer discovery, against a required 0.

Decision: with any count above zero, provider reconfiguration cannot produce a trustworthy result, because you cannot tell a bad mapping from a sick host. Here 3 > 0, so the order is fixed: capture diagnostics, restore the pair, then re-map the provider.

Capture the deadlock before you clear it #

A restart can clear blocked threads without removing what blocked them, so diagnostics come first:

  • Collect the gateway logs and thread diagnostics from the backup while the warning is active, using the gateway's own status and diagnostics pages.
  • Collect matching logs from the primary and one affected requesting gateway, with timestamps that overlap.
  • Record the upgrade path and the installed module versions on both members; a module left at an old version after a direct 8.1.32 to 8.3.4 jump is a real suspect.
  • Have the blocked thread stacks reviewed with the vendor's support to identify the affected subsystem and the applicable correction.

Plan the restart inside a maintenance window: taking down a member of the history-hosting pair interrupts history acquisition and everything downstream that reads it.

After the restart, verify three things before calling it recovered: the warning stays cleared, history requests resume, and the same blocked stacks do not return.

Restore the pair, then the connection bookkeeping #

The primary's running incoming connection from the backup and its disconnected redundancy peer are two different relationships, and the backup's missing outgoing connection is a configuration or state discrepancy to resolve; it is not by itself the cause of the deadlock. Checks:

  • Compare peer identities and destination addresses on both members.
  • Compare redundancy settings, gateway builds and module versions on both members.
  • Correct only a mismatch you have demonstrated.
  • If the diagnostics point to a software defect, take the applicable supported maintenance release and its upgrade procedure from the vendor; do not choose a release merely because it is newer.

After a failover or a forced restart, history can end up split across the pair; see what Ignition historian failover leaves behind in the second database before you assume records are missing.

Reconnect the remote history provider last #

With the pair healthy, the mapping work is short:

  • Open a fresh configuration view, select the intended gateway, and confirm the provider list refreshes from that gateway before you save. A list left over from a previously selected gateway is stale.
  • On the history host, check the service security settings on the configure tab so the requesting gateway is permitted to call remote services.
  • If discovery now works but queries return blank, verify the Tag Historian module is licensed on the gateway hosting the history; a missing module license there is a documented cause of blank remote history returns.
  • Accept recovery only when a known tag returns a known value and timestamp, not when the status string clears.

The intermittent behavior in the report, providers recording, stopping and later recovering, fits a service that a restart temporarily revives. That is why the deadlock capture matters more than another round of provider reconfiguration.

Questions asked

Why does the Gateway Network show running while the history provider says gateway not connected?

Running describes the transport between the two gateways. Provider discovery and history queries are services the remote gateway must actually process, and a gateway with deadlocked threads or a broken redundancy state can hold the connection open while those services are stuck. Treat the two statuses as separate checks and troubleshoot the hosting gateway, not the connection.

Why is the history provider list empty or showing another gateway's providers?

The list fills only after a fresh discovery request against the selected gateway completes. When that gateway cannot answer, the view stays empty or keeps the previous gateway's list, which is why it looks stale. Never save a provider mapping built from a stale list; reopen the configuration view and reselect the gateway once the host is healthy.

Should we restart the backup gateway to clear the deadlock?

Capture first, restart second. Collect the gateway logs and thread diagnostics while the deadlock warning is active, plus matching logs from the primary, and record the exact from-build and to-build numbers. A restart can clear the blocked threads without removing the cause, so afterwards verify the warning stays cleared and the same blocked stacks do not return.

Which upgrade path was involved in the reported case?

A direct jump from Ignition 8.1.32 (b2023091211) to 8.3.4 (b2026030309), with no intermediary upgrades. Include both build numbers, the module versions installed on each member of the redundant pair, and the redundancy configuration in anything you send the vendor, because a direct jump this wide multiplies the variables support has to rule out.

What if discovery works but the remote history query returns no data?

Check the gateway hosting the history, not the requesting one. Verify the Tag Historian module is licensed there, because a missing module license on the history-hosting gateway is a documented cause of remote queries returning blank, and confirm the underlying store holds the tags and time range you asked for. Prove it with one known tag, value and timestamp.

Keep reading

  • Troubleshooting

    Query tag history from a remote tag provider

    Yes: Tag Historian is one module with two jobs — storage and retrieval — so the gateway where system.tag.queryTagHistory runs must license it too. Install and license the module on the querying gateway; from 8.1 a read-only license covers a gateway that only consumes history. Prove the fix by matching row counts between the two gateways.

    Edge computing and IIoT

  • Selection guide

    Choosing an edge data path for periodic records

    For independent periodic samples, Tag Historian with Edge Sync Services is the reliable path and buffers 35 days locally. For one atomic row per collection cycle, use MQTT Transmission records landing in MQTT Recorder. For true transaction groups, license a standard gateway for that node.

    Edge computing and IIoT

  • Troubleshooting

    Fix Ignition Core Historian high CPU usage

    Gateway CPU above 80 percent that falls below 10 percent when only Core Historian is disabled is a historian storage fault, not a 30-tag acquisition load. Upgrade the 8.3.3 (b2026012009) gateway to 8.3.7 or newer, whose release notes fix Core Historian query performance degrading as data accumulated, and leave the partition interval and maintenance age as configured.

    Edge computing and IIoT