IMD SolutionComputing LibraryShop main store

How-to

Replaying buffered InfluxDB records to Azure IoT Hub over MQTT

Edge computing and IIoT Published 8 min read

Short answer

Yes — but not with the TICK stack alone. Telegraf's MQTT output forwards the live stream, not database history, and Kapacitor aggregates; run a small replay worker that queries InfluxDB after a durable (time, name) cursor and publishes each raw record to Azure IoT Hub at QoS 1, advancing the cursor only on acknowledgment. At 1 record/s arriving, publish at 5 records/s so a full 7-day buffer of 604,800 records clears in 42 hours.

On this page
  1. Are there TICK stack options that replay stored InfluxDB records to MQTT?
  2. How do you build the replay worker?
  3. How do you keep the replay from duplicating or skipping records?
  4. How fast must the replay publish to catch up?
  5. How long should the InfluxDB buffer keep records?
  6. What does Azure IoT Hub expect from the replay publisher?
  7. How do you prove the replay delivered every record?
  8. Questions asked

You can use InfluxDB as the store-and-forward buffer and republish every record to Azure IoT Hub, but not with the TICK stack's own plugins. Telegraf's MQTT output publishes metrics as they flow through the agent — it never queries the database back — and Kapacitor tasks aggregate. The reliable shape is a small replay worker that queries InfluxDB after a durable cursor and publishes each raw JSON record at QoS 1, advancing the cursor only on the hub's acknowledgment. With the source at one record per second, the replay must publish faster than that, or the backlog never moves.

Are there TICK stack options that replay stored InfluxDB records to MQTT? #

No — neither stock component does this job: Telegraf's [[outputs.mqtt]] publishes metrics as they flow through the agent, and Kapacitor tasks aggregate, so a dedicated replay worker beside them is the right build.

Telegraf's MQTT output, configured as its table in the agent's TOML file per the plugin's sample configuration, forwards whatever passes through the pipeline at that moment. Adding it next to your existing InfluxDB output gives you a live tee, not a buffer replay: nothing in that plugin reads history back out of the database, and it keeps no delivery position between runs.

Kapacitor earns its keep on alerts and windowed computation. A task that windows or aggregates emits derived points, so a raw database query and the task's output stop matching — exactly the mismatch that sinks these attempts, because the per-second JSON your downstream machine-learning path expects is no longer what arrives.

Decision: keep the working Telegraf-to-InfluxDB leg untouched, and put the replay in its own worker. It is the same store-and-forward shape you configure when the buffering device is an edge controller rather than a database: store-and-forward backfill configuration on an edge controller.

How do you build the replay worker? #

Build it as a small always-on service on the same edge host as Telegraf and InfluxDB: an eight-step loop that loads a cursor, queries strictly after it, publishes each raw record at QoS 1, and persists the cursor only after the hub acknowledges.

Run it on the same industrial mini PC that already collects the data, next to the agent and the database — the loop is one short query and a handful of publishes per second. Size that host for ingestion plus replay together, the way you would when sizing an edge PC for oversampled PLC data to InfluxDB.

  1. Leave ingestion alone: the device keeps publishing, Telegraf keeps consuming, and every JSON record keeps landing in InfluxDB exactly as it does today.
  2. Run the worker as a supervised service — a systemd unit or a container with a restart policy — so a crash or reboot resumes it without you.
  3. Load the delivery cursor, the last confirmed record time plus name, from a small file on persistent storage, persisted with write-to-temp-then-rename so power loss cannot tear it.
  4. Query the buffer strictly after the cursor, ordered by time then name, limited to a fixed batch of 500 rows.
  5. Publish each record's original JSON — datetime, name, value untouched — to the hub at QoS 1 and block until the publish acknowledgment returns.
  6. After each acknowledged record, advance the cursor to that record and persist it before publishing the next one.
  7. Loop once per second; a query returning fewer rows than the batch limit means the replay is caught up.
  8. Treat (datetime, name) as the deduplication key on the consuming side in Azure, because QoS 1 promises at-least-once, not exactly-once.
# Illustrative loop; these variable names are ours, not any product's.
cursor_time : string = load from cursor file   # RFC 3339, e.g. "2020-01-19T22:15:00Z"
cursor_name : string = load from cursor file   # e.g. "TempSens1"
batch_limit : int    = 500
publish_qos : int    = 1

every 1 s:
    rows = records ordered by time then name,
           strictly after (cursor_time, cursor_name), at most batch_limit rows
    for each row:
        publish row JSON at publish_qos        # blocks until the hub acknowledges
        cursor_time, cursor_name = row.time, row.name
        write cursor file, rename over the old one

How do you keep the replay from duplicating or skipping records? #

Track a durable cursor of the stored record time plus the name field, query strictly after that pair, and advance it only once the hub's publish acknowledgment for that record has come back — that is what removes both duplicates and gaps.

A query window that overlaps the last one — everything from now minus one second, every second — re-reads boundary records and duplicates them; a window that starts after a delayed write skips them. The strictly-after cursor closes both: overlap is impossible because each query begins past the last confirmed record, and a late write only waits until the next loop, one second later.

The name half of the cursor is the tiebreaker. Sensors can share a timestamp within the same second, and ordering by time then name is deterministic and resumable mid-stream. In InfluxDB 1.x a point is its measurement, tag set and timestamp, so two records from one sensor at the same instant collapse into one anyway — the pair still protects multi-sensor ties and every restart.

A crash between an acknowledgment and the cursor write republishes that record on restart: a duplicate, never a gap. That trade is deliberate, and the consumer's deduplication key from step 8 absorbs it.

How fast must the replay publish to catch up? #

Publish at 5 records per second: with the source still arriving at 1 record/s, that leaves 4 records/s of net drain, and a completely full 7-day buffer of 604,800 records clears in 42 hours.

The arithmetic, with this stream's arrival rate:

  • Arrival: 1 record/s × 86,400 s/day = 86,400 records/day.
  • Full 7-day buffer: 86,400 records/day × 7 days = 604,800 records.
  • Net drain at 5 records/s out, 1 record/s in: 5 − 1 = 4 records/s.
  • Worst-case drain: 604,800 records ÷ 4 records/s = 151,200 s = 42 h.
  • Equal rates: 1 record/s out and 1 record/s in leaves 1 − 1 = 0 records/s of drain, so the backlog never shrinks — not slowly, never.

Decision: publish at five times arrival for this stream. Any rate above arrival drains eventually, but five moves a worst-case week of data out in under two days while ingestion continues.

How long should the InfluxDB buffer keep records? #

Seven days is the right retention for this stream, because it covers a week-long outage and still leaves days of margin over the worst-case drain time computed above.

Retention has to outlast the longest outage you intend to replay plus the time the replay needs to drain it; set it through the database's retention setting — a retention policy on 1.x, bucket retention on 2.x. The buffer is the database's job, not the broker's: brokers forward, they do not keep queryable history, which is why does an MQTT broker store historic data you can query matters when someone proposes the broker as the buffer.

What does Azure IoT Hub expect from the replay publisher? #

Use one persistent MQTT connection at QoS 1 and size the send rate against your hub's documented device-to-cloud throttle; the 250,000-device concurrency figure in the quotas documentation is about connections, not your limiting number.

The standard doing the work here is MQTT's QoS 1, defined in the OASIS MQTT specification: an at-least-once agreement in which the client retries a publish until the broker answers with its acknowledgment — duplicates are possible, silent loss is not. The worker leans on exactly that semantics; do not publish the replay leg at QoS 0, which has no acknowledgment and leaves a loss window needing other cover, the problem covering the QoS 0 loss window in Sparkplug MQTT transmission works through.

The Azure IoT Hub quotas and throttling documentation is the page with the device-to-cloud send throttle for your tier, and that throttle at your provisioned unit count must sit above 5 records/s with margin. Two more figures from that documentation put the design in scale: the MQTT v5 connection profile, in preview, supports at most 250,000 concurrently connected devices at 25 units — a connection ceiling a single worker never approaches — and the device-to-cloud endpoint tops out at 32 partitions on basic and standard hubs, which is hub-side fan-in capacity, not a per-worker constraint.

The worker authenticates as one device identity, with a symmetric key or an X.509 certificate; certificate wiring has failure modes of its own, the ones fixing X.509 certificate auth to Azure IoT Hub over MQTT walks through. Decision: one connection, one identity, QoS 1, 5 records/s verified against the documented throttle.

How do you prove the replay delivered every record? #

Test it with 300 records of known values and timestamps: the worker must deliver exactly 300 unique messages, survive a kill mid-batch without loss, and emit JSON byte-identical to the stored records.

  1. Stop ingestion and insert 300 records you can predict, in the shape of your stream: {"datetime": "2020-01-19 22:15:00", "name": "TempSens1", "value": 25.9} and its neighbors.
  2. Start the worker and time it: 300 records ÷ 5 records/s = 60 s to drain.
  3. Count at the hub by unique (datetime, name): exactly 300. Total messages above 300 are duplicates; unique keys below 300 are loss, and that is a defect.
  4. Kill the worker mid-batch and restart it: still exactly 300 unique keys, with duplicates allowed only past the last persisted cursor.
  5. Compare every emitted message field-for-field against the stored record — datetime, name, value unchanged. This is the test an aggregation pipeline fails, and it is why the raw query lives in the worker.
  6. Restore ingestion and watch the loop: caught-up queries return one row per second, and the cursor advances with the source.

Ship the worker when all six pass. If step 5 mismatches, something between the query and the publish is transforming records, and that is a fault to fix, not a tolerance to tune.

Questions asked

Why can't Telegraf's MQTT output replay what InfluxDB already stored?

The output plugin fires on metrics as they pass through the agent's pipeline, so it publishes the live stream, not database history. To reuse it you would have to re-feed every buffered record into Telegraf as if it were newly sampled, and the plugin still keeps no durable delivery position, so the cursor problem lands back in your lap either way.

How does the arithmetic change with more sensors?

Both sides scale. Six sensors at one record per second each give 6 records/s arriving and a full seven-day buffer of 3,628,800 records; publishing at 30 records/s leaves a net drain of 24 records/s, and the full buffer clears in 151,200 seconds, or 42 hours again, because the five-to-one ratio is what fixes the drain time.

What happens to duplicate deliveries after a restart?

A crash between the hub's acknowledgment and the cursor write republishes at most the records after the last persisted cursor, so duplicates appear but records never go missing. Make the Azure consumer idempotent on the (datetime, name) pair, and a duplicate costs one insert that overwrites or drops, not a corrupted model input.

Should the worker keep one MQTT connection open or reconnect each cycle?

Keep one persistent connection. Reconnecting every second adds a TLS handshake and an authentication round trip to each loop, adds connection churn the hub must absorb, and buys nothing: the acknowledgment you need for the cursor arrives on the same open connection.

Keep reading

  • Selection guide

    Choosing an edge data path for periodic records

    For independent periodic samples, Tag Historian with Edge Sync Services is the reliable path and buffers 35 days locally. For one atomic row per collection cycle, use MQTT Transmission records landing in MQTT Recorder. For true transaction groups, license a standard gateway for that node.

    Edge computing and IIoT

  • Troubleshooting

    Why buffered Sparkplug B data is rejected after reconnect

    Replayed samples are dropped because the MQTT Publisher component in System Platform v2023 P01 sends them without the per-metric is_historical = true flag the Eclipse Sparkplug B specification defines for late data, so the broker treats old timestamps as suspect. No setting in that component sets the flag; keep MQTT for live values and recover the outage window through a separate backfill path.

    Edge computing and IIoT

  • Explainer

    MQTT to Kafka edge data pipeline without losing messages

    Keep MQTT on the sensor side of the plant, put the Kafka-class durable log on the site or cloud side of the uplink, and give the boundary one disk-backed store-and-forward buffer sized to the longest outage you will ride through. At 500 messages per second and an 8-hour outage, that queue holds 14.4 million messages and about 9.6 GB — set the ceiling above it and alarm on overflow.

    Edge computing and IIoT