Comparison
Choosing edge storage for high-rate vibration data
Short answer
Keep the CSV-in-ZIP vibration archive that already works at 10 kHz, and treat Parquet as a candidate format, not a capacity fix. Move only after the Parquet writer proves exact sample fidelity, sustained throughput at the real acquisition rate, and recoverable completed archives after power loss. A format change is an engineering change with its own acceptance test, and compression in either format never repairs precision lost upstream.
On this page
Keep the CSV-in-ZIP archive that already carries your 10 kHz vibration stream, and treat Parquet as a candidate, not a capacity fix. Concretely: at 10 kHz on one binary32 channel the storage floor is 40 kB/s (3.456 GB/day), text round-trips need 9 significant digits for binary32 and 17 for binary64, and a queue that returns to empty within one archive interval is the throughput pass mark. The proven path, typed floats converted to text at a controlled precision and compressed into bounded ZIP archives, has one known weakness: float-to-text precision has to be set deliberately. Parquet removes that text step by storing typed columns, but it adds a new writer, a new file layout and new failure modes. Change formats only after the candidate passes three tests on your hardware: exact sample fidelity against the values the serializer receives, sustained throughput at the real acquisition rate, and completed archives that survive power interruption. The numbers, the acceptance criteria and the per-case decision follow.
Two candidates, one place a sample can change #
Both formats run the same chain: acquisition buffer in RAM, serialization, compression, file write, durable media. A sample can only change at serialization, the step that turns each floating-point measurement into bytes. Everything after that step is lossless by construction: ZIP entry compression and Parquet's column encodings and codecs reproduce exactly the bytes they are given, so a compressor can neither repair a damaged sample nor damage a good one.
That collapses the format decision into three measurable properties: what the serializer writes (fidelity), how fast the chain sustains the real acquisition rate (throughput), and what an interrupted write costs (recovery). The table below scores both formats on those three points plus the read path.
Precision: the step that can alter a waveform #
The governing standard is IEEE 754. A binary32 sample carries a 24-bit significand, and its decimal text must carry 9 significant digits to read back as the identical float; binary64 carries 53 bits and needs 17. Emit fewer digits and some samples change, silently. A trend plot will not reveal it, and a spectrum computed from rounded data still looks plausible.
In the text path, precision lives in the CSV writer. In pandas, DataFrame.to_csv accepts a float_format argument, and its default is not a round-trip guarantee. Commit to one rule for waveform archives: text that round-trips, meaning 9 significant digits for binary32 and 17 for binary64, or the writer's shortest-round-trip mode, which emits the fewest digits that read back identically. Anything shorter is a recorded operating limitation, never a silent default.
The binary path removes the text step entirely: a Parquet FLOAT or DOUBLE column stores the IEEE bits directly. That removes the risk, not the duty to verify. Read one real capture back and compare sample for sample against the values the serializer received. The acceptance number is exact: max(abs(original - reconstructed)) = 0 whenever round-trip fidelity is required. Judge values, never a plotted waveform.
Throughput: derive the floor, then measure the writer #
No format stores below the binary payload, so derive the floor from the running configuration before comparing formats:
payload rate (bytes/s) = samples per second per channel × channels × bytes per sample
Worked example: one channel, binary32, 10 kHz #
Stated assumptions: single channel, 32-bit float, continuous acquisition, decimal prefixes with 1 kB = 1000 bytes.
- Binary floor: 10,000 samples/s × 4 bytes/sample = 40 kB/s.
- Daily floor: 40 kB/s × 86,400 s = 3.456 GB/day, the volume no format can go under.
- Uncompressed text at 9 significant digits: round-trip text for a signed float with its separator runs about 12 to 14 bytes, so take 13. That gives 10,000 × 13 = 130 kB/s, and 130 kB/s × 86,400 s = 11.232 GB/day before compression, about 3.25× the binary floor. Deflate shrinks that by a ratio only a measurement of your digit patterns will give you.
- 120 kHz upgrade case: twelve times the samples means twelve times the floor, 480 kB/s and 41.472 GB/day (480,000 bytes/s × 86,400 s), with uncompressed text near 1.56 MB/s. Compressed output does not scale with the sample rate; measure it at 120 kHz before believing any projection.
Retention runway falls out of the same numbers: free bytes ÷ daily bytes. At 41.472 GB/day, 500 GB of free space holds about 12 days (500 ÷ 41.472 ≈ 12.06) before pruning must run. That runway covers this stream only; slower process data retained elsewhere on the plant has its own capacity math, covered in sizing a self-hosted Node-RED InfluxDB Grafana historian.
What to measure and the pass mark #
- Read the real sample rate, channel count, sample type and duty cycle from the acquisition configuration.
- Measure uncompressed bytes over a representative capture, headers and metadata included.
- Time serialization, compression and the media write separately across at least one archive interval.
- Watch the capture queue. A queue that climbs without draining means sustained writer throughput is below the incoming rate, and a bigger buffer only delays the failure. A queue that spikes and empties means bursts, and a buffer sized to the largest observed spike absorbs them.
- Read completed archives back and check sample counts and sequence boundaries against the acquisition record.
Pass mark: at the real acquisition rate, the queue returns to empty within one archive interval, inside 60 s for 60-second archives, and stays there across a full day.
If the measured sustained rate misses that mark and no buffer closes the gap, the constraint is the writer box, not the format. That is the moment to specify a fanless industrial mini PC sized to the measured rate; re-encoding the data will not rescue a writer that cannot keep up.
Recovery: what a power cut is allowed to destroy #
A completed RAM buffer is not storage. Durability starts when the storage layer has flushed the bytes to persistent media and the application marks the archive complete. Before that point an interrupted write can leave a truncated file, a stale record or a half-updated index, which is why edge writers work in bounded, all-or-nothing capture units.
Both formats pay the same tax at the end of the file. A ZIP archive carries its directory at the end, so an archive cut off by power loss generally will not open, and repair tools are an emergency measure, not a design. Parquet writes its footer, which carries the file metadata, last, so a half-written Parquet file is typically unreadable too. Bounded archives cap the damage: at the worked example's rate, one 60-second interval holds 600,000 samples, about 2.4 MB of binary32, and that interval, not the day, is the worst case a power cut destroys.
Make each archive self-describing and self-checking: capture start time, sample rate, channel identity, units, sample count and the precision setting used, stored beside the samples. ZIP records a CRC-32 for each member, so a read-back that verifies the CRC and the manifest sample count proves the bytes arrived intact. Retention and pruning apply to completed records only. Acknowledge downstream delivery only after the archive is complete and durable, the same discipline that keeps an MQTT to Kafka edge data pipeline without losing messages.
One safety note: run power-cut tests on a bench copy of the writer. Never interrupt storage on a box that also serves a control or safety function.
The formats side by side #
| Decision point | CSV in ZIP, the baseline | Parquet, the candidate |
|---|---|---|
| Where precision is fixed | at float-to-text conversion, by the writer's precision setting | at the column type (FLOAT, DOUBLE), no text step |
| Round-trip rule | 9 significant digits for binary32, 17 for binary64, or shortest-round-trip mode | exact when the decoded column type matches the acquisition type; verify once |
| Uncompressed rate, 10 kHz one-channel example | about 130 kB/s of text | 40 kB/s binary floor |
| Compression | ZIP entry compression, lossless; ratio measured, never assumed | column encodings plus codec, lossless; ratio measured, never assumed |
| Interrupted write | archive missing its end-of-file directory will not open | file missing its footer will not open |
| Bounded damage window | the archive interval, e.g. 60 s is about 2.4 MB at the example rate | the row-group or file interval you configure |
| Read path | any text tool after decompression | a Parquet reader library on every consumer |
| Change risk | none while it holds the working rate | a new writer, layout and set of failure modes |
Feed both writers the same recorded waveform with its metadata, and compare full continuous coverage against full continuous coverage. Comparing one format's complete capture against the other's selected chunks confuses storage efficiency with discarded data.
The switch test on a recorded waveform #
Run the candidate against the baseline before touching the live writer.
- Record one representative capture, continuous coverage including a startup and a shutdown, and feed the identical bytes to both writers.
- Fidelity: read both outputs back and require max(abs(original - reconstructed)) = 0 against the serializer's input values.
- Soak: run the Parquet writer at the real rate for a full day. Pass means the queue returns to empty within one archive interval throughout, with peak memory inside the box's budget.
- Interruption: cut power mid-write on the bench copy, ten times at random points in the archive interval. Every archive marked complete must open, pass its CRC and carry its manifest sample count; incomplete files are quarantined, never served to a reader.
- Cut over only after all three pass, and keep the CSV-in-ZIP path for one retention period as the rollback.
The decision, case by case #
- Working 10 kHz system without storage pressure: keep CSV-in-ZIP. Set round-trip precision once, record the setting, stop.
- Storage pressure at 10 kHz: prune completed archives by retention and check the measured write rate first. A format change is an engineering change, not a capacity fix.
- 120 kHz upgrade: budget the floor at 480 kB/s and 41.472 GB/day, then soak the candidate at 120 kHz on the actual hardware. Do not extrapolate a 10 kHz compression ratio to it.
- Parquet passes fidelity, soak and interruption: switch, with the proven path held as rollback for one retention period.
Questions asked
Can existing CSV-in-ZIP archives be converted to Parquet later?
Yes, and that is the safe order of operations. Keep collecting in the proven format, convert old archives offline on a workstation rather than on the edge box, and verify each converted file before retiring the original: same sample count, same values read back, manifest intact. Conversion is a batch job with unlimited retries; the live writer has neither.
Will Parquet files actually be smaller than ZIP-compressed CSV?
Not automatically. Both paths compress losslessly, so neither can beat the binary floor of samples per second times bytes per sample, and neither owns a guaranteed ratio over decimal text. Column encodings reward long runs of similar values and lose ground on noisy, high-entropy digit patterns. The only honest number is the same waveform through both writers on your hardware.
What precision is needed for 64-bit vibration samples?
Seventeen significant digits. IEEE 754 binary64 carries a 53-bit significand, and 17 significant decimal digits are the minimum for text to read back as the identical double; binary32 needs 9. Prefer a shortest-round-trip output mode over a fixed number of decimal places, because a fixed count truncates large values and pads small ones while neither behavior shows on a plot.
Should startup and shutdown captures be saved with the rest?
Keep them, and name them explicitly in the retention rules. Startup and shutdown intervals carry the load transients and abnormal contact that steady-state windows miss, and they are the first casualty of any save-only-selected-chunks policy. If storage pressure forces selection, select among completed archives after capture instead of dropping intervals at acquisition time, where the loss is unrecoverable.