IMD SolutionComputing LibraryShop main store

Troubleshooting

Fixing extra characters in a PLC MQTT JSON payload

Edge computing and IIoT Published 8 min read

Short answer

The extra characters come from publishing a UTF-16 WSTRING to a subscriber that reads UTF-8, and from sending SIZEOF bytes of the allocated buffer instead of the text. Convert the finished JSON with JSON.Stu.ConvertUTF16toUTF8 into a STRING(2000), then publish TO_UDINT(JSON.Stu.StrLenA(ADR(utf8JsonData))) bytes. Keep the MQTT 5 properties; they declare UTF-8 but convert nothing.

On this page
  1. Why does a JSON payload from my PLC arrive with extra characters?
  2. Is the problem the string encoding or the payload length?
  3. How do I convert the JSON wide string to UTF-8 before publishing?
  4. What payload length should the MQTT publish call send?
  5. What does the MQTT payload format indicator promise about my payload?
  6. Does trimming the wide string fix the garbage bytes?
  7. How do I verify the corrected payload at the subscriber?
  8. Should I fix the publisher or the subscriber instead?
  9. Questions asked

A JSON document that builds correctly in the PLC can still reach the MQTT broker wrapped in characters no subscriber recognizes. The JSON is not at fault. The publish call sends a UTF-16 wide string straight out of a WSTRING to clients that read UTF-8, and it passes SIZEOF, the entire allocated buffer, as the payload length, so the wire carries a zero byte after every character plus everything stored behind the text. Two changes on the publisher fix it: convert the finished JSON to UTF-8 in a byte-oriented string, and publish exactly the byte count that conversion produced. This page separates the two defects, shows the corrected publish call, explains what the MQTT 5 payload format indicator actually promises, and gives the byte-count check that proves the payload is clean.

Why does a JSON payload from my PLC arrive with extra characters? #

The JSON is not the problem: your publish call sends a UTF-16 wide string straight to a subscriber that reads UTF-8, and it sends the entire allocated buffer instead of the length of the actual text, so every character arrives followed by a zero byte.

In the program behind this symptom, wsJsonData is declared WSTRING(1000) and JSONByteArrayWriter writes the finished document into it as wide characters. A wide string stores UTF-16, two bytes per character, so ASCII text travels as the character byte followed by 0x00. A subscriber that parses UTF-8 renders the character, then hits a byte that is invalid in UTF-8 and prints an unrecognized-character mark in its place. The publish itself succeeds: the client connects, the broker accepts, the subscription receives. That is why the fault appears at the subscriber instead of as an error in the runtime, and why the JSON logic can be perfectly correct while the payload looks hopeless.

The bytes, not the document, are the defect. If your tags reach MQTT through an OPC UA server rather than a builder in the runtime, the same byte rules apply once the payload is on the wire; the article on bridging OPC UA PLC data to MQTT with an edge PC covers that collection path.

Is the problem the string encoding or the payload length? #

Check both defects, because either one alone produces garbage: the encoding mismatch puts an invalid byte after every character, while the oversize length appends everything the buffer holds behind the closing brace.

  • Encoding defect. wsJsonData holds UTF-16. Published as-is, every ASCII character arrives as two bytes, the second being 0x00, which a UTF-8 client cannot decode and displays as an extra or unrecognized character.
  • Length defect. udiPayloadSize := SIZEOF(wsJsonData) passes the storage of the declaration, the 2000 bytes of character storage that WSTRING(1000) reserves, no matter how many bytes the JSON occupies, so the broker also receives everything behind the text.

The symptom tells you which defect you are looking at: garbage between the characters means encoding, and a long tail of noise after the closing brace means length. A payload built the way this one is has both, which is why the client shows characters it cannot recognize from the first brace to the last byte.

The length defect is not unique to strings: publishing a REAL value straight from its memory bytes produces the same unrecognized ASCII, which the page on resolving ASCII garbage when an MQTT publish sends REAL values walks through.

How do I convert the JSON wide string to UTF-8 before publishing? #

Convert the finished JSON into a byte-oriented STRING(2000) with JSON.Stu.ConvertUTF16toUTF8, then publish from that converted string; the function takes the source address, the destination address and the destination buffer size, and writes the UTF-8 bytes there.

Declare the destination next to the wide string:

VAR
    wsJsonData   : WSTRING(1000); // built by JSONBuilder and JSONByteArrayWriter
    utf8JsonData : STRING(2000);  // UTF-8 copy that goes to the broker
END_VAR

Run the conversion after the writer has finished filling wsJsonData. The third argument is the capacity of the destination, and it bounds what the conversion may write:

JSON.Stu.ConvertUTF16toUTF8(
    ADR(wsJsonData),
    ADR(utf8JsonData),
    SIZEOF(utf8JsonData)
);

Then point the publish at the converted buffer:

mqttPublish(
    mqttClient    := mqttClient,
    pbPayload     := ADR(utf8JsonData),
    udiPayloadSize := TO_UDINT(JSON.Stu.StrLenA(ADR(utf8JsonData))),
    wsTopicName   := topic,
    mQTTPublishProperties := mqttPublishProperties
);

Nothing upstream changes. JSONBuilder and JSONByteArrayWriter still build the document into wsJsonData exactly as before, the jsonArrayWriter call still uses pwData := ADR(wsJsonData) with udiSize := SIZEOF(wsJsonData) as its capacity, and mqttClient still connects with eMQTTVersion := MQTT.MQTT_VERSION.V5. Keep mqttPublishProperties with bPayloadFormatIndicator := 1 and wsContentType := "application/json"; the section below explains what those two declarations promise.

Wide strings deserve the same attention on other links: carrying long WSTRING tags over OPC UA takes its own configuration, which the page on configuring WSTRING tags over 254 characters in TIA Portal OPC UA describes.

What payload length should the MQTT publish call send? #

Publish exactly the byte length of the converted text: JSON.Stu.StrLenA returns that length, so the argument becomes TO_UDINT(JSON.Stu.StrLenA(ADR(utf8JsonData))), while SIZEOF keeps reporting the allocated buffer no matter how short the JSON is.

Worked example, using the document this program builds around Key1, Key2 and Key3, assuming the writer emits it compactly with no added spaces: {"Key1":"Value1","Key2":{"Key3":1234}}. Count the characters: 38. All are ASCII, so the UTF-8 copy is 38 bytes, one per character. The same 38 characters in the WSTRING occupy 76 bytes, because UTF-16 stores each one in two bytes. The original publish sends the full 2000-byte buffer, so the subscriber receives 76 bytes of JSON followed by 1924 bytes (2000 - 76) of trailing memory that no client can use.

After the conversion, JSON.Stu.StrLenA(ADR(utf8JsonData)) measures the actual bytes and returns 38 for this document. TO_UDINT casts that count to the UDINT type of udiPayloadSize, and the wire now carries the document and nothing else. Of the two candidates, only this length measures the payload; SIZEOF measures the declaration.

What does the MQTT payload format indicator promise about my payload? #

A payload format indicator of 1 tells the broker and every subscriber that this publish carries a well-formed UTF-8 string, so the original program declared UTF-8 while delivering UTF-16 bytes, a contradiction a validating client treats as malformed.

MQTT version 5 added this property, and the program sets it through bPayloadFormatIndicator := 1 inside mqttPublishProperties. The value 1 declares UTF-8 string data and obliges the publisher to send well-formed UTF-8; wsContentType := "application/json" then tells applications how to parse the bytes once they are decoded. Neither property converts or checks anything inside the PLC: by the time the broker has the packet, the bytes are whatever the publisher handed over.

That is exactly what happened here. The declaration said UTF-8, the bytes were UTF-16, and subscribers that honor the declaration showed the mismatch as characters they cannot recognize. After the conversion fix, the declaration becomes true and both properties can stay as they are. A payload that is genuinely binary rather than UTF-8 text is the case for leaving the indicator at 0; JSON on MQTT is not that case.

Does trimming the wide string fix the garbage bytes? #

Trimming does not help: MSU.StrTrimW strips blank characters from a wide string but leaves it UTF-16, and the publish call still sends the full allocation, so neither the encoding defect nor the length defect changes.

The original program calls MSU.StrTrimW(pString := ADR(wsJsonData)) after the writer finishes, which trims blanks inside wsJsonData and nothing else. The bytes stay UTF-16, so the interleaved invalid bytes remain; the length argument stays SIZEOF, so the trailing memory remains. Trimming treats what the symptom looks like rather than either cause.

Once the conversion and the byte-count length are in place, the trim is redundant: StrLenA measures the actual converted text, so trailing blanks would already fall outside the published range. The decision: drop the reliance on trimming, fix the encoding with the conversion, and fix the length with StrLenA.

How do I verify the corrected payload at the subscriber? #

Verify by counting bytes at the subscriber, not by eyeballing characters: with the corrected call the broker must deliver exactly the length JSON.Stu.StrLenA returned, 38 bytes for the example document, and the client must parse the JSON with nothing trailing it.

  1. Publish once and record the length argument the call sent: TO_UDINT(JSON.Stu.StrLenA(ADR(utf8JsonData))).
  2. At the subscriber, check that the received payload byte count equals that recorded value, not the full buffer the old call pushed.
  3. Show the raw payload: text characters only, no 0x00 bytes interleaved, and nothing after the closing brace.
  4. Change a tag value and publish again; the new value must appear in the next payload. If the first sample keeps repeating while the tag moves, the fault has moved to the update path, which the page on an MQTT Engine JSON payload not updating on the second publish walks through from the subscriber side.

A payload that passes all four checks is clean: well-formed UTF-8, correct length, and current values.

Should I fix the publisher or the subscriber instead? #

Fix the publisher: convert to UTF-8 and send the StrLenA byte count. Pointing one subscriber at UTF-16 can make a single client parse the payload, but every historian, dashboard and script added later expects UTF-8 JSON, so the publisher-side conversion is the durable repair.

Reading the payload as UTF-16 works for exactly the client you reconfigure, and it breaks the declaration the payload format indicator makes. Every consumer added afterwards, a historian, a dashboard, or an MQTT to Kafka edge data pipeline without losing messages stage, assumes well-formed UTF-8 JSON. Converting at the publisher repairs all of them at once and keeps the declaration honest.

The decision, case by case: encoding defect, convert with JSON.Stu.ConvertUTF16toUTF8; length defect, publish TO_UDINT(JSON.Stu.StrLenA(ADR(utf8JsonData))) bytes; trimming, remove it; properties, keep bPayloadFormatIndicator := 1 and wsContentType := "application/json"; subscriber, leave it on UTF-8.

Questions asked

Why do my tag changes never reach the broker after the first publish?

In this program the JSON tree is built once, inside IF xFirst THEN, and xFirst is set FALSE right after, so JSONByteArrayWriter keeps serializing the first sample no matter what the tag does later. Rebuild the tree, or update the relevant keys, on every cycle before you run the conversion and the publish; then confirm at the subscriber that a changed tag reaches the next payload.

How large should the UTF-8 destination buffer be?

Size it for your largest document, not for this example. The conversion call takes the destination capacity as its third argument, SIZEOF(utf8JsonData) or 2000 bytes here, and writes the UTF-8 copy within that bound. Note that UTF-8 can need up to three bytes for a character UTF-16 stores in two, so text with non-ASCII characters grows when converted; check the converted length of your biggest payload against the buffer.

Do the payload format indicator and content type work on MQTT 3.1.1?

No. The payload format indicator and the content type are MQTT version 5 properties, and the client in this program requests version 5 by connecting with eMQTTVersion := MQTT.MQTT_VERSION.V5. An MQTT 3.1.1 publish has no place to carry them, so on older brokers the payload must be correct UTF-8 on its own, which the conversion already guarantees.

Who checks that the payload really is well-formed UTF-8?

The obligation sits with you, the publisher: an indicator of 1 declares well-formed UTF-8, and nothing in the conversion chain enforces it for you. If the conversion did not run, or the length came from SIZEOF again, the packet goes out contradicting its own declaration, which is exactly the symptom this page started with. Test the payload bytes, as the verification steps above do.

Keep reading

  • How-to

    Bridging OPC UA PLC data to MQTT with an edge PC

    Keep the OPC UA server on the PLC side and give MQTT publishing its own role on a separate edge PC that acts as the OPC UA client and MQTT publisher. Commission the two links independently — first a verified tag read, then a verified subscriber message — and only then widen the tag set.

    Edge computing and IIoT

  • How-to

    Sending PLC data to an IoT hub without custom software

    You do not need custom software to move PLC data to an IoT hub: a configurable edge package with an S7, Modbus TCP or EtherNet/IP driver, a channel mapping and an MQTT publisher replaces hand-written acquisition code. Verify two things before you commit: the driver covers the protocol your controller actually speaks, and the publisher covers the destination's authentication and message format. Software still does the work — you configure it instead of writing it.

    Edge computing and IIoT

  • How-to

    Harden a Node-RED OPC UA server exposing PLC tags

    A default Node-RED OPC UA server accepts any client that can reach its port until three changes land: set message security mode to SignAndEncrypt with the Basic256Sha256 policy, remove mode None and anonymous login, and admit clients only through application-certificate trust lists. Those three changes are the whole decision; the rest is execution.

    Edge computing and IIoT

SCADA MQTT points blank with nested JSON payloads