Time Series

Retention and downsampling on InfluxDB OSS, and why the detector cannot eat a downsampled stream

The test cell's bucket is filling, and the question the series deferred five times is forced. A year of full-rate vibration does not fit on a 40 GB disk, so the raw stream gets a short window and the long-term record gets built from rollups. The catch is that the anomaly detector from issues 06 through 08 reads the exact transient structure that downsampling throws away, so the long-term memory has to be the feature table, not a coarsened waveform.

Since issue 05 the test cell has written full-rate vibration and five process variables into a single InfluxDB OSS bucket on the 40 GB Hetzner CX22, and every issue since has deferred the question of what happens when the disk fills. Issue 09 forces it, because the bucket is filling on a measurable schedule. Two assets streaming an accelerometer channel at two kilohertz plus their process variables produce on the order of a gigabyte a day after compression, which fills the usable disk in roughly three weeks, so retention is no longer optional. The naive fix, a short retention period on the raw bucket, drops history the plant will want the day a bearing fails. The standard fix, downsampling raw data into coarser rollup buckets, is lossy in exactly the dimension that matters, because the anomaly detector from issues 06 through 08 computes its features from the transient shape of the waveform and a one-minute mean keeps none of it. The resolution is to separate the two records the system actually needs. The raw waveform is a short forensic buffer with a seven-day retention period. The long-term memory is the per-window feature vector the edge already computes, which is small enough to keep at full event cadence for years. Downsampled rollups serve the dashboards. A flagged anomaly triggers a pin that copies its raw window into a permanent forensic bucket before the retention sweep can delete it, because a retention policy is a destructive operation on a timer and the window you needed is always the one that just expired. The cost line is unchanged at $5.50 a month plus an optional storage box for cold export, and the reader retrofit window has still not closed.