How-to
Turn an MQTT topic into InfluxDB fields with Telegraf
Short answer
Telegraf reads /temperature/boiler/room3 with payload 21.7 as a metric carrying a topic tag and a value field. A five-processor chain (regex, converter, regex, regex, pivot) renames the measurement, builds location=room3 and pivots the middle segment into the field key; one Starlark processor does the same in eight lines. Either way the output line is temperature,location=room3 boiler=21.7 with the original timestamp.
On this page
- How do I build an InfluxDB line from an MQTT topic in Telegraf?
- What does Telegraf hand the processors before any mapping?
- Which processors rename the measurement and add the location tag?
- How does pivot turn a topic segment into the field key?
- Is Starlark a shorter way to do the same mapping?
- What happens to topics with more or fewer segments?
- How do I check the metric after each stage?
- Does the bucket tag choose the InfluxDB bucket?
- Questions asked
Telegraf can rebuild the message inside its own processor chain, so the publisher keeps sending /temperature/boiler/room3 with the bare payload 21.7 and every other subscriber keeps working. The target line is temperature,location=room3 boiler=21.7 carrying the original timestamp, and two configurations produce it: a five-processor chain of regex, converter, regex, regex and pivot, or a single Starlark processor of eight lines. Standardize on Starlark unless a site rule forbids embedded scripts; keep the chain as the configuration-only fallback. Both are built below, with a worked trace you can reproduce from one test message and the two places where the versions disagree stated as decisions.
How do I build an InfluxDB line from an MQTT topic in Telegraf? #
Two configurations do it: a five-processor chain of regex, converter, regex, regex and pivot, or one Starlark processor that rebuilds the metric in eight lines with no ordering dependencies. Both take /temperature/boiler/room3 with payload 21.7 and emit temperature,location=room3 boiler=21.7.
The path is publisher to MQTT broker to Telegraf MQTT consumer to processors to InfluxDB output. The consumer subscribes to your configured topics and parses each payload with the data format you set, so mapping work starts from a metric that already exists. Because other software consumes the same publication, the payload stays untouched and every change belongs on the Telegraf side. The choice between built-in processors and a script is the same trade-off you meet whenever a payload arrives in a shape the parsers dislike, for example gjson versus Starlark for Telegraf JSON arrays. Default to Starlark here: fewer moving parts and an exact output line. Use the chain only where configuration must stay script-free.
What does Telegraf hand the processors before any mapping? #
One single-field metric per message: the input plugin's own measurement name, a topic tag holding the full /temperature/boiler/room3, and a value field holding the parsed payload, all timestamped in nanoseconds.
Topic and payload travel as separate parts of the publication, so there is nothing to split at a space. The payload's parsed type carries through the whole pipeline: 21.7 arrives as a float, myvalue as a string, and nothing downstream changes one into the other. For the worked example used throughout, the input line is:
mqtt_consumer,topic=/temperature/boiler/room3 value=21.7 1767225600000000000
If no metric appears at all, the fault is delivery rather than mapping; start with why a device cannot publish MQTT to a broker on your PC. If the line appears with topic and value, delivery and parsing are proven and everything that follows is transformation.
Which processors rename the measurement and add the location tag? #
A regex at order = 1 copies the first topic segment into a tag, the converter at order = 2 promotes that tag to the measurement name, and a regex at order = 4 writes the third segment into location. The sequence is fixed, because later blocks read tags that earlier blocks created.
The complete chain:
[[processors.regex]]
order = 1
[[processors.regex.tags]]
key = "topic"
pattern = "^/([^/]*)/.*"
replacement = "$1"
result_key = "bucket"
[[processors.converter]]
order = 2
[processors.converter.tags]
measurement = ["bucket"]
[[processors.regex]]
order = 3
[[processors.regex.tags]]
key = "topic"
pattern = "^/[^/]+/([^/]+).*"
replacement = "$1"
result_key = "garbage"
[[processors.regex]]
order = 4
[[processors.regex.tags]]
key = "topic"
pattern = "^/[^/]+/[^/]+/(.*)"
replacement = "$1"
result_key = "location"
[[processors.pivot]]
order = 5
tag_key = "garbage"
value_key = "value"
Read it one block at a time. Order 1 pulls the measurement name out of the topic into a temporary tag named bucket; order 2 spends that tag as the new measurement name; order 3 parks the field name in a tag called garbage; order 4 writes the location into location. The pivot at order 5 is the piece that builds the field key, and it needs its two inputs already in place.
How does pivot turn a topic segment into the field key? #
It rotates one tag into the field key: with tag_key = "garbage" and value_key = "value", the pivot at order = 5 takes the second topic segment, parked there by the regex at order = 3, as the new field name and the payload as that field's value.
The pivot processor exists to rotate streams of single-valued metrics into one compact multi-field metric, and that rotation is exactly what a topic-derived field key needs: the future key must already sit in a tag, which is why order = 3 exists at all. The tag it rotates is consumed; the others are not. After order = 5 the worked metric reads:
temperature,location=room3,topic=/temperature/boiler/room3 boiler=21.7 1767225600000000000
The topic tag rode through untouched. To reach the exact target line, drop leftovers with the tag filtering the output plugin accepts, which removes topic and any spent temporary tag in one place.
Is Starlark a shorter way to do the same mapping? #
Yes, one [[processors.starlark]] block splits the topic, builds a new metric named by the first segment, sets location from the third, and copies the payload into a field named by the second, keeping the original timestamp.
[[processors.starlark]]
source = '''
def apply(metric):
topic = metric.tags["topic"] # string, e.g. "/temperature/boiler/room3"
s = topic.split('/') # list of strings; s[0] is empty
m = Metric(s[1]) # new metric named by segment 1
m.tags["location"] = s[3]
m.fields[s[2]] = metric.fields["value"]
m.time = metric.time # nanosecond epoch, copied unchanged
return m
'''
The new metric carries only what the script sets, so the output is exactly temperature,location=room3 boiler=21.7 with no tag cleanup step. The script is Python-syntax and whitespace-sensitive, and fixing Starlark indentation errors in Telegraf scripts is the first move when the processor refuses to load. The value of m.time is a nanosecond epoch; converting Starlark epoch time to a date belongs to display tooling, not to this mapping.
What happens to topics with more or fewer segments? #
The two versions disagree by design: the chain's location pattern keeps everything after the second segment, so /a/b/c/d produces a location tag of c/d, while the Starlark split uses only s[3] and silently drops the fourth segment.
Empty segments behave differently as well: the first regex accepts an empty leading segment because its capture allows zero characters, while the Starlark split would build a metric with an empty measurement name. Neither implementation validates the topic. Decide from the topic contract: with exactly three segments guaranteed, use either version; with a possible tail that must be kept whole, use the chain; with anything nonconforming, reject or reroute it upstream of the mapping rather than letting it write malformed points.
How do I check the metric after each stage? #
Swap the InfluxDB output for a file output that prints to standard output, publish one message with a known topic and payload, and read what appears; a stage-by-stage trace finds the first wrong pattern in under a minute.
- Point Telegraf at a file output on standard output instead of the database while testing, so every processed line is visible.
- Publish one test message against a broker you control, the same setup used to run a secure Mosquitto MQTT lab broker with Node-RED, with topic /temperature/boiler/room3 and payload 21.7.
- Check the split arithmetic before trusting any pattern: three slashes yield four elements at indexes 0 through 3, the leading slash makes s[0] empty, so s[1] = temperature, s[2] = boiler and s[3] = room3.
- Match the printed lines against the stages:
| Stage | Effect on the worked metric |
|---|---|
| Input | mqtt_consumer,topic=/temperature/boiler/room3 value=21.7 1767225600000000000 |
| order = 1 | adds tag bucket=temperature |
| order = 2 | measurement becomes temperature |
| order = 3 | adds tag garbage=boiler |
| order = 4 | adds tag location=room3 |
| order = 5 | writes field boiler=21.7 |
- Compare the final line with the target temperature,location=room3 boiler=21.7 1767225600000000000; the timestamp is the published time in nanoseconds, 1767225600 s x 10^9 = 1767225600000000000 ns, and it must match exactly.
- Put the InfluxDB output back and confirm the same line lands in the bucket the output names.
Does the bucket tag choose the InfluxDB bucket? #
No, bucket is just the name given the temporary extraction tag; the measurement name and the destination bucket are independent settings, and bucket selection happens in the InfluxDB output configuration, never in a processor.
Renaming the measurement to temperature does not send the point to a bucket called temperature. If different measurements must land in different buckets, that routing belongs in the output configuration on the Telegraf side, decided per deployment. Rename the temporary tag or keep it; it has no routing effect, and any leftover temporary tags come out with the same output tag filtering that removes topic.
Questions asked
Why does the field arrive as a quoted string in InfluxDB?
The payload parser typed it that way. A numeric payload such as 21.7 parses to a float and stores boiler=21.7; a non-numeric payload such as myvalue stays a string and renders as a quoted value in line protocol. Neither the pivot rotation nor the Starlark field copy converts between types, so correct the type at the input data format or with a conversion step before the write.
Can the MQTT consumer split the topic itself?
Yes. The consumer's topic_parsing option divides an incoming topic into named tags at the input stage, which replaces the three regex extractions. You still need the converter to promote one tag to the measurement name and the pivot to rotate another into the field key, so the back half of the chain remains.
What does an empty topic segment do?
It produces an empty string element after the split. The chain's first regex tolerates it because its capture allows zero characters, but the Starlark script would name the new measurement with an empty string. Neither implementation validates the topic, so enforce the expected segment count upstream and reject malformed paths before they reach the mapping.
Can one Telegraf instance map several topic schemas?
Yes. Give each schema its own processor blocks and restrict each block to the topic pattern it should touch, using the metric filtering that Telegraf plugins accept. Match on the topic tag, because before the measurement is renamed there is nothing else schema-specific to filter on.
Does the mapping change what other subscribers receive?
No. Every transformation runs inside Telegraf after the broker has delivered the message. The publisher keeps sending the same topic with the bare payload, so any other software reading that publication from the broker sees exactly what it saw before you added the mapping.