Troubleshooting
Trace an edge gateway MQTT connection failure to Azure Event Grid
Short answer
A desktop MQTT client that connects and publishes with thumbprint authentication clears only that endpoint, certificate and permission combination on that host. When the gateway fails against the same Event Grid namespace, test four stages in order on the gateway itself: network path and DNS to TCP 8883, the TLS handshake with both certificate directions and the clock, the CONNECT identity, and publish authorization on the exact topic. The first stage that fails names the fix.
On this page
A desktop MQTT client that connects and publishes to the Event Grid namespace with thumbprint authentication clears exactly one combination: that endpoint, that certificate pair, that identity, those topic permissions, from that host and its route. It clears nothing on the gateway. Split the gateway's failure into four stages — path, TLS, CONNECT, publish — test them in order on the gateway itself, and the first stage that fails hands you the fix. When a test client works and the gateway does not, the fault sits almost always in the gateway's own route, certificate files, clock, CONNECT identity or topic authorization, not in the namespace.
What the working test session proves #
A passing test session settles four facts: the endpoint serves MQTT over TLS on 8883; the server certificate chain validates against the test host's trust store; the certificate registered on the Event Grid client resource authenticates a client; and that client may publish its topics. None of those facts transfers to the gateway, which runs its own route, its own files, its own clock and its own identity. Compare the two clients field by field before changing anything in the publishing logic:
| Input | What the passing test session settles | What the gateway must still prove |
|---|---|---|
| Host name and port | Endpoint resolves and answers TLS on 8883 from the test host's route | Its own DNS, route and firewall to the same host on 8883 |
| Server trust | Test host's trust store accepted the server chain | Its configured trust file must hold a root that validates that chain |
| Client certificate | The registered thumbprint matched the test client's certificate | Its certificate file's SHA-256 must equal the registered Primary Thumbprint |
| Private key | The test client proved possession of the matching key | Its key file must pair with its own certificate file |
| CONNECT identity | Client ID and username accepted for that client | Its client ID must be unique and its username must carry the authentication name |
| Topic permission | That topic was publishable for that client | Its client needs the binding for the exact topic string |
Run the stages in order; each one gates the next:
- Path: resolve the namespace host name and open TCP 8883 from the gateway.
- TLS: complete the handshake with the trust file, certificate pair and clock correct.
- CONNECT: get the session accepted with the identity fields the namespace expects.
- Publish: write one payload to the exact topic and read it back from a permitted subscriber.
Keep session identities distinct while you test. A broker attaches a session to its client ID, so the gateway and the test client must not share one: two clients on the same ID evict each other in turns, and the symptom reads as random disconnects rather than an authentication fault.
Path first: DNS and TCP 8883 #
The Event Grid MQTT endpoint is TLS-only; there is no plain-text port beside 8883 to fall back on. Test on the gateway, in this order: interface state and default route; DNS resolution of the namespace host name; an outbound TCP connection to port 8883. A desk workstation proving the endpoint reachable does not clear the gateway, because a cellular modem or an OT VLAN often carries firewall rules the desk subnet never sees. If TCP cannot complete, stop there: no certificate or topic setting matters yet. Some gateways surface their own link-not-ready condition as an error from the runtime's last-error value right after the connect attempt; read that value before touching certificates. The same stage order for a client that never arrives at its broker is laid out in fixing a PLC MQTT client that never reaches your broker.
TLS: two certificate checks and the clock #
Mutual TLS runs two independent checks, and a correct thumbprint satisfies only one. Outbound, the gateway validates the namespace's server certificate against the trust file it is configured to use. Inbound, Event Grid authenticates the gateway's client certificate, either signed by a CA the namespace trusts or self-signed and matched by thumbprint — the portal flow registers the certificate's SHA-256 value in the Primary Thumbprint field of the client resource. The split between a certificate that authenticates you and trust material that validates the broker is the same one described in X.509 certificate authentication to Azure IoT Hub over MQTT. A self-signed client certificate authenticates the client; it never becomes server trust material.
Check, in this order:
- Thumbprint: compute the fingerprint of the deployed file with
openssl x509 -in client.crt -noout -fingerprint -sha256and compare it character for character with the registered Primary Thumbprint. Illustrative example: the resource registersAB:24:7E:...and the file computes9F:7C:02:...— the file was regenerated; redeploy the registered pair or re-register the new thumbprint. - Key pair: the key file must be the private key of the certificate file. For an RSA pair,
openssl x509 -in client.crt -noout -modulus | openssl sha256andopenssl rsa -in client.key -noout -modulus | openssl sha256must print the same hash. - Trust file (ca.pem in this setup): it must hold a root that validates the server chain the endpoint presents today; a bundle carried over from an earlier project fails here. Take current public CA material from the certificate authority itself.
- TLS version: the client must offer TLS 1.2 or newer — the Azure service floor; older versions fail before any certificate is judged.
Certificate validity makes the clock a first-class suspect. Worked example: a self-signed certificate generated on 11 April 2024 with a 3,650-day life is valid from 2024-04-11 until 2024-04-11 + 3,650 days. Ten calendar years from that date is 3,652 days, because 2028 and 2032 each contribute a 29 February, so 3,650 days ends on 2034-04-09. A gateway showing the correct date sits inside the window. A gateway whose real-time clock reset to 1 December 2019 sits 1,593 days before its own certificate becomes valid — 366 days through 2020's leap February, three 365-day years, then 132 days to 11 April 2024 — and the handshake dies with a not-yet-valid verdict on a certificate that is registered perfectly. Sync the gateway clock to a time source and the same files pass.
One safety rule: treat the client private key as a live credential. Generate it on the gateway or move it over a controlled path, never through chat or a shared folder, and re-register the thumbprint if the key has ever left your control.
CONNECT: the identity the namespace expects #
A completed handshake moves the fault to the CONNECT packet, and three fields decide it. The username must carry the client's authentication name as registered on the namespace; with certificate authentication the password field stays empty. The client ID identifies the session, not the credential — keep it unique per client and stable across reconnects. The protocol version must be one the namespace accepts, MQTT 3.1.1 or 5.0; a gateway pinned to 3.1.1 is fine, and 5.0 buys reason codes that name failures precisely. A keepalive of 60 s is a normal choice.
The decisive read at this stage is on the broker side. Enable the MQTT authentication logs — Diagnostic settings, under Monitoring, on the Event Grid namespace page in the Azure portal — and the log separates never-arrived, which sends you back to the path and TLS stages, from arrived-and-refused, which names the field that failed. If TCP completes but the session never opens, work through MQTT connect fails when TCP works: diagnose the handshake.
Publish: permission on the exact topic string #
A CONNECT accepted is not a publish granted. Event Grid attaches publish rights to clients through permission bindings on topic spaces, and the topic template must match the topic string the gateway writes, character for character — devices/topic1 in the failing setup. Publish one fixed payload at QoS 1 to that exact string and read it back from a subscriber permitted to receive it, closing the last leg exactly as MQTT clients publishing and subscribing through the broker describes. If the subscriber stays silent while the CONNECT succeeded, correct the topic template or the permission binding, or fix a drifted topic string — a trailing slash or a changed capital is enough. An accepted CONNECT next to a rejected publish in the namespace logs is a template problem, not a certificate one.
The decision at each stage #
- DNS fails on the gateway: fix the gateway's resolver or route. Nothing later in the chain is testable until a lookup succeeds.
- TCP to 8883 cannot connect: open outbound TCP 8883 toward the namespace on the gateway's own route and retest from the gateway, not from a desk PC.
- TCP connects, TLS alerts: let the failure name the file — trust file for the server chain, thumbprint or key pair for the client side, clock for validity windows — fix that one thing, and rerun.
- A gateway that cannot negotiate TLS 1.2, or that loses its clock at every power cycle, is finished as a data publisher: replace it with a fanless industrial mini PC and rerun the same four stages on the new unit.
- TLS completes, CONNECT refused: set the username to the registered authentication name, re-register the thumbprint of the file actually deployed, remove client ID collisions, and confirm the verdict in the namespace's authentication logs.
- CONNECT accepted, subscriber silent: the topic string misses the topic template or the binding is missing — publish
devices/topic1at QoS 1, read it back, and correct the template or binding to match.
Work the stages in order and the failing stage carries its own fix; the passing test client only ever proved that the cloud side was not the problem.
Questions asked
Which MQTT protocol versions and QoS levels does Event Grid accept?
Event Grid namespaces accept MQTT 3.1.1 and MQTT 5.0, and they deliver at QoS 0 and QoS 1. QoS 2 is not supported, so a client hard-set to QoS 2 fails at publish even with certificates, identity and permissions all correct, which reads as a topic-permission problem in the logs. Keep publishing clients at QoS 1 when a lost message matters.
How do I rotate the gateway certificate without locking it out?
Register the new certificate's SHA-256 thumbprint in the Secondary Thumbprint field while the current certificate still works. Deploy the new certificate and key pair to the gateway, confirm it connects, then move the secondary value into Primary Thumbprint. Replacing the file and the registered thumbprint in a single step leaves no working credential on record, so one wrong character in the fingerprint locks the gateway out until it is corrected.
What do the Event Grid authentication logs actually show?
Each CONNECT appears with the client authentication name, an accept or reject verdict, and a reason. The reasons split into authentication failures such as a thumbprint mismatch, an unknown authentication name or a username that does not carry the authentication name, and authorization failures on a publish or subscribe. Read the reason before changing gateway files: a rejected publish after an accepted CONNECT points at the topic template or permission binding, not at the certificates.
Why do the gateway and the test client keep disconnecting each other?
They are sharing one client ID. A broker keeps one session per client ID, so each connect evicts the other, and the pattern looks like unexplained drops every few minutes, often near the keepalive boundary. Give the test client its own ID or disconnect it entirely before judging the gateway's connection stability.