Troubleshooting
Rebirth fails after switching Sparkplug MQTT brokers
Short answer
After a broker switch, Sparkplug telemetry arriving on the new connection proves only the inbound path. A rebirth request must leave through the client session that is actually connected on the enabled MQTT Server; when the module mishandles multiple server connections, the NCMD publish fails as client offline and no births return. Verify the replacement connection once, re-establish the birth baseline, and run a module build with the documented multi-server rebirth correction.
On this page
- Where each message must travel after a switch
- Rule out the plain connection fault first
- Read the log sequence: data in, command out failed
- Re-establish the birth baseline on the enabled connection
- Working the switch on Ignition 8.1.4 with MQTT Engine 4.0.7
- The build and license state that close it
- The decision for each outcome
- Questions asked
When Sparkplug telemetry keeps arriving after a broker switch but recovery never starts, the fault is on the outbound command path, not the data path. A rebirth only works if it leaves through the client session that is actually connected on the newly enabled MQTT Server connection and lands on spBv1.0/[groupID]/NCMD/[nodeID]; incoming data proves nothing about that path. On Ignition 8.1.4 with the MQTT Engine module at 4.0.7, the reported pattern — data arriving for a node without a birth, a rebirth request logged, then a publish failure naming the NCMD topic as client offline — matches a documented defect in rebirth handling with multiple MQTT Server configurations, corrected after 4.0.7. Verify the replacement connection once, re-establish the birth baseline, and run a module build that carries the correction.
Where each message must travel after a switch #
Sparkplug B keeps context separate from change. A node publishes NBIRTH once per connection with every metric definition on spBv1.0/[groupID]/NBIRTH/[nodeID]; a device does the same on the DBIRTH topic. Only after those births do NDATA and DDATA updates mean anything to the host. When a host starts without that context, or loses it, the recovery defined by the specification is a rebirth request: the host publishes a node command carrying a single Boolean metric, and the edge node, which subscribes to its own NCMD topic, answers by sending its births again.
| Message | Topic pattern | Direction across the switch |
|---|---|---|
| Node birth | spBv1.0/[groupID]/NBIRTH/[nodeID] | Node to host; must precede NDATA |
| Device birth | spBv1.0/[groupID]/DBIRTH/[nodeID]/[deviceID] | Device to host; must precede DDATA |
| Node and device data | spBv1.0/[groupID]/NDATA/[nodeID] and the DDATA topic | Node to host; proves the inbound path only |
| Rebirth command | spBv1.0/[groupID]/NCMD/[nodeID] | Host to node; the recovery path |
Each configured MQTT Server connection is its own client session with its own state on the host. Broker clustering replicates messages between brokers; it does not merge those sessions. Disabling one connection and enabling another is therefore a session handoff the module must perform deliberately: connect on the new server, notice the missing birth context, and publish the rebirth through that same connected client. The reported case fails at exactly that last step.
Rule out the plain connection fault first #
Work down this list on the replacement connection before touching Sparkplug logic:
- Read the connection state at the moment data arrives; enabled is not the same as connected.
- Confirm the gateway can resolve and reach the endpoint: name resolution, routing, listener port, and the transport and TLS settings the listener expects.
- Confirm the broker accepts the session: credentials, then permissions to publish to the group NCMD topic and subscribe to the node topics. A broker that drops the client at connect is a different fault, walked through in why a broker rejects your Sparkplug client over its STATE will.
- A client that cannot reach the broker at all — DNS, route, or firewall — is its own diagnosis; start with fixing a PLC MQTT client that never reaches your broker.
- Apply the discriminator: a genuinely down connection stops the data. Here data continues, which is what makes this fault look healthy when it is not.
Read the log sequence: data in, command out failed #
The reported gateway logged, in order: node and device data arriving for an edge node that had not sent a birth on the new connection, then a decision to request rebirth from [groupID]/[nodeID], then this line:
Read each line for what it rules out:
- The birth warnings mean the host is holding data it cannot map. They read like edge node misbehavior, but the node is innocent: nobody asked it to rebuild.
- The rebirth line means the module chose the documented recovery.
- The offline-client line means the publish failed at the selected client before MQTT carried anything. An independent subscriber watching the new broker confirms the strongest fact in the case: no NCMD ever appears on it.
- Switching back to the first server connection locks onto the stream immediately, which clears the node, both brokers, and the cluster, and leaves the second connection session handling as the suspect.
Re-establish the birth baseline on the enabled connection #
- Run the test with exactly one server connection enabled. On the original connection, record steady state: NBIRTH, DBIRTH, and changing data values.
- Connect an independent MQTT client to the same broker, subscribed to spBv1.0/[groupID]/#, with rights to see the command and birth topics. It is your ground truth for what actually leaves the host.
- Perform the switch: disable the original server connection, enable the replacement carrying the same group and node.
- Trigger a rebirth deliberately: the host rebirth action, or a node command with the Boolean rebirth metric set to true on spBv1.0/[groupID]/NCMD/[nodeID]. If a hand-built command seems to vanish, check its payload structure first; why the Rebirth metric arrives without its Sparkplug alias covers the alias mechanics.
- Watch the subscriber. A healthy handoff shows one NCMD on the enabled broker, then NBIRTH followed by the DBIRTHs.
- If the host logs the rebirth request and the subscriber still sees no NCMD, stop adjusting the broker. The publish path inside the connection handling is faulty, and the fix is a build change, not a network change.
A command published through a connection other than the enabled one tests nothing: one client, one broker, one observer.
Working the switch on Ignition 8.1.4 with MQTT Engine 4.0.7 #
Inputs, as reported for this case: a gateway on Ignition 8.1.4, host module MQTT Engine 4.0.7, two MQTT Server configurations (A the original, B the replacement) whose brokers sit in one cluster, and one edge node with one device beneath it. The instruments are the gateway log plus one independent subscriber on broker B authorized for spBv1.0/[groupID]/#. Sparkplug B sets the expectation: births precede data on every connection, and recovery is one NCMD carrying the rebirth metric on spBv1.0/[groupID]/NCMD/[nodeID].
The check counts what crosses broker B after the switch. For a successful handoff, the host must deliver exactly 1 rebirth command to the node and receive at least 1 NBIRTH and 1 DBIRTH back. Data messages arriving while the delivered-command count stays at 0 mean the command path failed, whatever the network looks like.
- After the switch (A disabled, B enabled): the log records inbound NDATA and DDATA, so messages in: at least 2.
- Births in: 0. The log states the data arrived without a birth.
- Commands attempted: 1. The rebirth request is logged.
- Commands delivered on B: 0. The offline-client failure names the NCMD topic, and the subscriber saw no NCMD.
- Tally: delivered commands (0) below attempted (1) while inbound data (at least 2 messages) keeps flowing, so the outbound path failed and the inbound path is alive.
- Control: switching back to server A syncs immediately with 0 commands needed, clearing the node and both brokers.
Result: the rebirth never left the host through connection B; the module held no live session on that client at publish time.
Decision: data flowing in with 0 of 1 commands delivered is the signature. Verify endpoint, credentials, and broker acceptance for B once, to exclude a plain disconnect. If they pass and the NCMD publish still fails while data arrives, stop touching the network, deploy a module build with the multi-server rebirth correction, and repeat this test with the subscriber watching.
The build and license state that close it #
Version 4.0.7 release notes list one relevant addition: sending a rebirth request when DDATA arrives before DBIRTH. The reported gateway already ran that build, so the correction covers a missing device birth on a single connection, not a switch between server connections. The module vendor later documented that rebirth requests were not sent properly in certain scenarios with multiple MQTT Server configurations and resolved it in a nightly build published after that release.
Decision: run a build that includes the multi-server rebirth correction, record the exact module build in the change notes, and re-run the switch test before trusting the handoff in production.
License belongs in the same check. An expired trial stopping the module is a demo limit, not part of this fault; in the reported case, re-activating the trial resynced the stream, consistent with the module rebuilding its state on activation. A gateway that must run continuously needs a permanent module license: verify the active license on the deployed target instead of leaning on re-activation.
The decision for each outcome #
- Data in, rebirth logged, offline-client failure on the NCMD topic, and no NCMD on the broker: an outbound session-state defect. Verify the endpoint once, then move to a build with the documented multi-server rebirth correction and repeat the switch test.
- No data and no rebirth after the switch: the inbound path is gone. Fix reachability, credentials, or broker acceptance before touching Sparkplug.
- NCMD visible on the enabled broker but no births returning: the host side is fine, and the fault sits in the edge node command subscription or its connection to that broker.
- Design rule: any architecture that moves a host between brokers — failover to a redundant second broker is the common case, patterned in PLC MQTT failover to a redundant second broker — must be validated by performing the switch with a subscriber on the command topic. When the second broker is reached through a bridge rather than a directly configured server, birth certificates have their own documented failure mode; spBv1.0 edge node birth certificate fix for an MQTT broker bridge covers that variant.
Questions asked
Does broker clustering make two MQTT Server connections interchangeable?
No. Clustering replicates messages between brokers, but each configured server connection keeps its own client session and connection state on the host side. Disabling one connection and enabling another is a session handoff the module must handle explicitly, and the documented defect lives in exactly that handling. Test the two configurations as separate connections even when both brokers belong to one cluster.
Can I trigger the rebirth manually instead of waiting for the host?
Yes. From any client connected to the enabled broker with rights on the group command topics, publish a node command containing the Boolean rebirth metric set to true to spBv1.0/[groupID]/NCMD/[nodeID]. A healthy edge node answers by publishing its node birth followed by its device births on that same broker. If the births arrive but the host stays stuck, the fault is on the host side, not the node.
Which module build resolves the multi-server rebirth fault?
Version 4.0.7 already added a rebirth request when device data arrives before the device birth, and the reported case ran exactly that build, so that correction does not cover a switch between server connections. The multi-server rebirth fault was resolved in a nightly build shipped after that release. Check the module release notes for that correction before you upgrade.
Why does data keep arriving if the log says the client is offline?
The offline report names the publishing client at the moment of the rebirth attempt, while the incoming stream is still delivered on the enabled connection. In the reported case a separate MQTT subscriber confirmed that no command ever reached the broker, which is why the stream looks healthy while the recovery path is dead. Telemetry arriving is therefore never proof that a broker switch succeeded.