IMD SolutionComputing LibraryShop main store

Troubleshooting

Why a broker rejects your Sparkplug client over its STATE will

Edge computing and IIoT Published 6 min read

Short answer

The broker refuses the CONNECT because the client's MQTT Last Will is declared on a STATE topic its ACL does not authorize for publish, and wills are checked at connect time before any PUBLISH. Grant publish rights on that one STATE topic or place an intermediate broker between client and cloud; changing subscriptions does nothing.

On this page
  1. What the broker is refusing
  2. Confirm the will is the denial
  3. Why STATE is guarded so strictly
  4. Three ways through
  5. Worked example: one SCADA consumer against a core-only STATE ACL
  6. Bridge details that decide success
  7. Questions asked

Your client is being refused at CONNECT, before it has published anything. A Sparkplug host application puts its Last Will on its STATE topic inside the CONNECT packet, and the broker checks that will topic against the client's publish permissions at connect time. When the access control list reserves STATE for core servers only, the connect is refused, and no change to subscriptions, payloads or QoS will alter that. Two fixes are real: grant that identity publish rights on that one STATE topic, or put an intermediate broker between the client and the strict cloud broker so its will is never presented there. Which one you choose is decided by one question: can every edge node on the broker select which host it obeys? If any node cannot, keep the ACL closed and bridge.

What the broker is refusing #

An MQTT Last Will is declared as part of the CONNECT exchange. The broker stores it and publishes it only if the client later disappears without disconnecting cleanly. Because the broker is committing itself to publish on that topic in the future, it runs the topic past the client's publish permissions at connect time. A will aimed at a topic the identity may not publish to fails the whole connect.

That is what makes the symptoms misleading. The client reconnects in a loop, its credentials are correct, and no application message has ever been sent. In short:

  • More subscriptions change nothing: the failing check is a publish check on one topic.
  • Payload and encoding fixes change nothing: no PUBLISH packet has left the client.
  • A different client identifier helps only if that identity carries different publish rights.

Confirm the will is the denial #

Work the failure in this order before changing anything:

  • Correlate one connection attempt, by timestamp and client identifier, with the broker's rejection record.
  • Record the authenticated identity, the configured host ID and the exact will topic as the CONNECT declared it, from broker diagnostics or a packet capture, never derived from a display name.
  • Pull the effective ACL for that identity, explicit denies included, and compare the will topic against the publish grants; the shape such grants take is shown in access-control configuration for Sparkplug B.
  • Confirm the will topic sits in the STATE namespace your installation reserves for its core servers.
  • If the denial names authentication, a connection limit or anything other than a topic, stop: that is a different fault with a different fix.

Why STATE is guarded so strictly #

STATE is a JSON message a host application publishes to announce that it is online; edge nodes subscribe to STATE and use it to decide whether anyone is listening. The Sparkplug specification, section 1.4.1, puts it as a duty: "Sparkplug Host Applications MUST publish STATE messages denoting their online and offline status."

An issue raised against the specification (eclipse-sparkplug issue 531) argues the requirement belongs to the Primary Host Application alone, which matches the operational notes: a designated primary application publishes STATE, everyone else subscribes. Spec wording will move at spec speed, and no wording change rewrites an installed driver's CONNECT packet.

The guard on this broker exists for a concrete plant-floor reason. Some of the edge nodes cannot be configured to obey only named servers, and the site does not want them publishing while the core servers are offline. So STATE publish rights stop at the core servers, and every additional will on a STATE topic is refused on purpose. Edge nodes publishing through a host outage is not a cosmetic fault: it breaks the discipline that data flows only while a host is present to receive it.

Three ways through #

  • Grant publish on that one STATE topic. The clean fix when the server is a legitimate host on this broker and every edge node can choose which host it obeys, or when you want the nodes to obey it. The cost is real: this server's availability now influences edge-node publishing, and any node that cannot filter reacts to it.

  • Stop the client publishing STATE. Only if the installed release documents a switch. The Geo SCADA Sparkplug driver exposes none in the affected release, and a feature request is open with the vendor, so treat this path as closed until the vendor documents it. Changing the SCADA Host ID changes which topic the will declares; it does not stop the publish, and it only helps if the ACL happens to permit the new name.

  • Put an intermediate broker in front of the cloud. The server connects locally, where the ACL is yours and the will is authorized. A separate bridge client holds the cloud credentials, declares its own will on a topic the cloud permits, and forwards only the traffic the application needs. Run that broker on a fanless mini PC in the plant network; the local broker setup is covered in run a secure Mosquitto MQTT lab broker with Node-RED, and whether a broker belongs inside the controller or on separate hardware is its own decision, covered in running an MQTT broker in the PLC or on a separate device.

Worked example: one SCADA consumer against a core-only STATE ACL #

Inputs, each from the reported installation:

  • One extra Sparkplug consumer: a customer's SCADA server whose driver always publishes STATE and always declares its will there.
  • A cloud broker ACL that grants STATE publish to the core servers only.
  • Edge nodes that cannot be configured to obey only named servers.
  • No documented switch in the installed driver release to stop STATE publishing.

The rule in words: a broker accepts a CONNECT only if the identity may publish the will topic the packet declares, and an MQTT client declares exactly one will. An edge node that cannot choose a host obeys every STATE message that reaches it.

The arithmetic. Call the number of core servers N; this installation runs more than one, so N is at least 2. Take the usual redundant pair, N = 2:

  • STATE publishers authorized today: N = 2.
  • Publishers if the SCADA server is granted: N + 1 = 3.
  • Online STATE publishers during a full core outage, as designed: 0.
  • Online publishers in that same outage with the grant, SCADA server still connected: 1.

The edge nodes that cannot choose a host read that 1 as a host present and keep publishing through the outage, the exact behavior the ACL was written to prevent. Decision: with even one unfiltering edge node on the broker, the correct number of new publish grants on the cloud ACL is 0. Bridge the server through an intermediate broker, keep its STATE local, and let the bridge declare the only will the cloud ever sees. If every edge node can select its host by ID, the decision flips: 1 grant, on that single STATE topic, is the fix.

Bridge details that decide success #

  • The bridge's CONNECT carries its own will. Point it at a topic the cloud ACL permits, or you have moved the failure instead of fixing it.
  • Do not forward the SCADA server's STATE toward the cloud, and do not route core STATE toward local edge nodes in a way that lets them read this server as an authoritative host.
  • Dropping forwarded STATE messages does nothing about the bridge's own will; that will is declared at the bridge's connect and checked there.
  • Clear any retained STATE the server published during testing, and make sure no message can return by a second route and undo the filter.
  • Prove it: both connects accepted, a capture of the cloud-side CONNECT showing no will on a forbidden topic, and a hard kill of the SCADA server showing no STATE from it in the cloud.

Questions asked

Can a Sparkplug host publish STATE without declaring a will?

No. The will is what becomes the offline STATE if the host disappears without a clean disconnect, so a driver that implements STATE declares its will on the same topic. STATE without a will is not a documented option in any release covered here; the real choices are authorizing that one topic or moving the connection behind a broker that accepts the will.

Would granting publish on the whole Sparkplug namespace be a simpler fix?

It is simpler and wrong. This client needs publish rights on exactly one topic, its own STATE topic; everything else it does is subscribe. A namespace-wide grant does not change what this client sends, but a compromised identity could then inject births and data anywhere, and any STATE it publishes is obeyed by edge nodes that cannot choose a host.

Does the Sparkplug v3.0.0 STATE format change affect this?

It is a separate check worth doing at the same time. The STATE message format changed with Sparkplug B v3.0.0, and consumer support follows the spec: MQTT Engine reads the new format from release 4.0.14. A host on the new format and a consumer pinned before its supporting release will misread host availability, so confirm releases on both ends before touching the ACL or the bridge.

Why can't the broker accept the connect and just drop the will later?

Because the will is not a message that flows during the session; it sits inside the broker until the client disappears. A filter on published traffic never sees it. The MQTT permission model has no accept-but-suppress state for a will, so the broker's choices at CONNECT are authorize the topic or refuse the connection, which is why the ACL is the control point.

Keep reading

  • How-to

    Bridging OPC UA PLC data to MQTT with an edge PC

    Keep the OPC UA server on the PLC side and give MQTT publishing its own role on a separate edge PC that acts as the OPC UA client and MQTT publisher. Commission the two links independently — first a verified tag read, then a verified subscriber message — and only then widen the tag set.

    Edge computing and IIoT

  • Explainer

    Exporting Sparkplug B out of Geo SCADA to a historian

    No modified DDK driver will push Sparkplug B out of Geo SCADA, because the driver layer is pull-only in the outbound direction. Build the publisher on the Client API Sample DataFeeder pattern, which watches about 100,000 points per instance; for millions of points, use HSI or a database-level connector.

    Edge computing and IIoT

  • Troubleshooting

    Fix a PLC MQTT client that never reaches your broker

    Most PLC MQTT failures sit in one of three stages: the broker listener, the network path to the port the client is actually configured for, or the client's own publish logic. Test in that order; a connect timeout points to the first two, a silent publish to the third.

    Edge computing and IIoT