IMD SolutionComputing LibraryShop main store

Troubleshooting

Fixing an edge gateway MQTT TLS private key load failure

Edge computing and IIoT Published 5 min read

Short answer

The errors error:0906D06C PEM routines:PEM_read_bio:no start line and error:140B0009 SSL routines:SSL_CTX_use_PrivateKey_file mean the gateway cannot parse its own private-key file, so the failure is local and happens before the broker sees anything. On a Flexy 205 running firmware 15.0s1 the confirmed correction is the 15.0s2 firmware upgrade with the same certificate files, not new keys or AWS IoT policy edits.

On this page
  1. What the three log lines say
  2. Confirm it is the 15.0s1 firmware case
  3. Apply the correction
  4. Other causes of the same PEM error
  5. Why the AWS IoT policy is not involved
  6. The decision for each case
  7. Questions asked

Error:0906D06C (PEM routines:PEM_read_bio:no start line) followed by error:140B0009 (SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib) means the gateway could not read its own private-key file, so the MQTT connection died before any TLS handshake or broker authorization. On an Ewon Flexy 205 running firmware 15.0s1 this is the reported key-file handling fault in that firmware, and the confirmed correction is the 15.0s2 firmware upgrade with the script and certificate files left unchanged. Regenerating keys or editing AWS IoT policies misses, because this failure never reaches AWS.

What the three log lines say #

The log names the failing file first: Error: Unable to load client key file "/opt/ewon/usr/device-private.pem.key". The script asks for that file through the keyfile parameter, set just before the connect call:

MQTT "OPEN", device_id$, mqtt_broker$
MQTT "SETPARAM", "Port", "8883"
MQTT "SETPARAM", "cafile", "/usr/AWS_CA.crt"
MQTT "SETPARAM", "certfile", "/usr/device-certificate.pem.crt"
MQTT "SETPARAM", "keyfile", "/usr/device-private.pem.key"
MQTT "CONNECT"
ErrorReturned% = GETSYS PRG, "LSTERR"

device_id$ and mqtt_broker$ are the script's own string variables for the client ID and the broker endpoint. Decoded in order:

  • error:0906D06C:PEM routines:PEM_read_bio:no start line — the PEM parser scanned the key file and never found the boundary line that must open it. RFC 7468, which defines the text encoding for these files, requires a PEM object to start with a -----BEGIN----- line; that boundary is the "start line" the parser reports as missing.
  • error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib — the TLS context then had no parsed key to install, so loading the private key into the session failed.
  • BASIC [ERROR] Operation failed (28) 186 : MQTT "CONNECT" — the 28 is what GETSYS PRG, "LSTERR" returns when the MQTT "CONNECT" operation fails, whatever stage broke.

The sequence is local first: parse the key, load it into the SSL context, run the handshake, then the broker authenticates the certificate and evaluates policy. A failure in the first step ends the connection with the broker never contacted.

Confirm it is the 15.0s1 firmware case #

Four checks settle it:

  • The device is a Flexy 205 reporting firmware 15.0s1.
  • The log carries the three lines above, with the key named under /opt/ewon/usr/.
  • A key file copied from an identical unit that connects without error produces the same failure, so the key's contents are not the variable.
  • The same CA file, certificate and key connect to the same AWS endpoint from a workstation, so the certificate set and the policy work outside the gateway.

Size check, worked from one reported case: the generated key file measured 1,706 bytes. Suppose the copy read back from the gateway measures 1,760 bytes — 1,760 − 1,706 = 54 bytes of growth — and its first line no longer reads -----BEGIN RSA PRIVATE KEY-----. A rewritten header reproduces error:0906D06C exactly, and on 15.0s1 the rewrite happens on the device, which is why no re-upload corrects it.

Apply the correction #

  1. Download firmware 15.0s2 for the Flexy 205 from the HMS Networks support site; firmware comes from the manufacturer or its distributors, nowhere else.
  2. Upgrade the gateway following the vendor's documented update procedure.
  3. Upload the three TLS files again — the CA to /usr/AWS_CA.crt, the certificate to /usr/device-certificate.pem.crt, the key to /usr/device-private.pem.key — with binary-mode transfer.
  4. Re-run the same MQTT sequence from the script above, unchanged.
  5. Read the log: the key-file line and both OpenSSL errors must be gone, then confirm a publish on a topic the existing policy allows.

Move the key to the gateway over a direct, restricted session rather than a shared staging folder. If a read-back size check still differs from the source after the upgrade, redo the transfer in binary mode before suspecting anything else.

Other causes of the same PEM error #

On gateways that are not on 15.0s1, the same error string has three usual origins:

  • Key format: the file must open with -----BEGIN RSA PRIVATE KEY----- (PKCS#1) or -----BEGIN PRIVATE KEY----- (PKCS#8). OpenSSL's key reader does not parse the OpenSSH format ssh-keygen writes by default.
  • Transfer mode: text-mode FTP rewrites line endings and can clip the header line the parser scans for. Use binary mode.
  • Storage damage: re-download the key from the gateway and compare byte counts with the local copy. Matching sizes and a correct first line rule out the file and point back at firmware or format.

Why the AWS IoT policy is not involved #

The broker evaluates a policy only after a certificate has authenticated on a completed TLS handshake. This failure happens before the handshake starts, so no change to connect, publish or subscribe permissions can alter the outcome.

Rejection that appears only after the handshake — certificate trust, registration, thumbprints — is a different fault, and the steps in fixing X.509 certificate auth to Azure IoT Hub over MQTT cover that stage. A client that never reaches the broker at all, with timeouts and no OpenSSL lines, is a reachability problem; start with fixing a PLC MQTT client that never reaches your broker.

The decision for each case #

  • Flexy 205 on firmware 15.0s1 with these three log lines: upgrade to 15.0s2 and keep the script and all three files unchanged.
  • Key opens with the OpenSSH header: regenerate the pair as PKCS#1 or PKCS#8 PEM and issue the certificate for the new key.
  • Read-back size differs from the source after a binary-mode upload on corrected firmware: redo the transfer; a second mismatch means the storage path is altering files.
  • No OpenSSL lines and pure timeouts: reachability, not key loading.

And if this gateway was already scheduled for replacement rather than another firmware cycle, a fanless mini PC can take over the same AWS IoT publish job.

Questions asked

Does BASIC error 28 always mean the WAN interface is not ready?

No. The 28 comes from GETSYS PRG "LSTERR" after the MQTT CONNECT call and only says the operation failed. The sample script printed a WAN-not-ready message for that value, which pointed troubleshooting at the network. The MQTT log entries printed alongside it, with the two OpenSSL errors, are what identify the private-key load as the failing stage.

Why does the error name /opt/ewon/usr/device-private.pem.key when the script sets /usr/device-private.pem.key?

It is the same file. The SETPARAM keyfile entry uses the short /usr path, and the firmware reports the failed load under the device's full path /opt/ewon/usr/. The file name is identical in both messages, so no path correction is needed in the script.

My private key starts with BEGIN OPENSSH PRIVATE KEY. Does that cause this error?

It can. OpenSSL's private-key reader accepts PKCS 1 keys, which begin BEGIN RSA PRIVATE KEY, and PKCS 8 keys, which begin BEGIN PRIVATE KEY. It does not read the OpenSSH format that ssh-keygen writes by default. Regenerate the pair as PEM and issue the device certificate for the new key.

Should I widen the AWS IoT policy while troubleshooting this error?

No. The policy is evaluated only after a certificate authenticates on a completed TLS handshake, and this failure happens before the handshake starts. Fix the key loading first. Revisit permissions only if a later attempt connects and then has a publish or subscribe rejected.

How should the private key be handled on the gateway?

Treat it as a per-device secret. Move it to the gateway over a direct, restricted session instead of shared folders, keep FTP accounts limited to the people who need them, and delete workstation copies once the connection test passes. If a key may have been exposed, deactivate that certificate at the broker and issue a replacement pair.

Keep reading

  • Troubleshooting

    Fix CODESYS MQTT TLS peer verification handshake failure

    Enabling peer verification in the MQTT client's TLS context fails because the controller has no trust anchor for the broker's certificate chain. Import the issuing CA certificate into the runtime's trusted certificate area through the security screen; the client identity .pfx does not provide that trust. A TCP_Init error is an earlier socket-layer fault, so check endpoint settings before touching certificates.

    Edge computing and IIoT

  • Troubleshooting

    Fixing X.509 certificate auth to Azure IoT Hub over MQTT

    The controller fails because the selected .key file is only half an X.509 credential: TLS client authentication must present the device certificate and its chain, so the key passes file validation and the handshake still has nothing to send. Convert the test certificates to PEM, import the leaf-plus-chain with its matching key as one pair, make the hub's device ID or thumbprint match, and verify with a real MQTT connection.

    Edge computing and IIoT

  • Explainer

    Securing ctrlX MQTT data paths to AWS or GCP

    Yes — Node-RED plus Mosquitto can carry ctrlX Data Layer values under X.509 authentication and TLS 1.2, but each of the three connections (publisher to broker, device to broker, uploader to cloud) needs its own trust and client-certificate configuration. DeviceBridge joins only when the chosen AWS or GCP service's protocol calls for it.

    Edge computing and IIoT