Time-Series Data at the Edge: What to Store, What to Send, What to Drop

A sensor sampling once a second produces around 31 million readings a year. Multiply by a fleet and the volume becomes substantial, and the uncomfortable truth about most deployments is that almost none of it is ever read. It is transmitted, stored, paid for, and eventually deleted on a retention schedule without a human or a query ever touching it.

That is not automatically wrong — some data is insurance, valuable precisely because you cannot predict when you will need it. But the decision is usually made by default rather than deliberately, and the default is to send everything because storage feels cheap. It is cheap per gigabyte and expensive per fleet-year, which is a different calculation.

A more useful framing is to sort readings into three categories at the device. The first is data that changes a decision: a threshold crossed, a state transition, an anomaly. This should be transmitted immediately and reliably, and it is usually a tiny fraction of the total. The second is data that establishes context: the readings around an event, which are worthless alone but essential for diagnosing the event. This is worth buffering locally and sending only when something interesting happens near it. The third is everything else, which is worth aggregating rather than transmitting.

Aggregation at the edge is where most of the saving lives. A minute of one-second samples reduced to a minimum, maximum, mean and count preserves nearly everything an operator actually asks of it while shipping a fraction of the bytes. The information lost is genuine but specific: you can no longer reconstruct the exact shape of that minute. Whether that matters is a question the application can answer, and answering it deliberately is the entire point.

Local retention deserves the same care. A device with a few megabytes of flash can hold a surprising amount of aggregated history, which turns a connectivity outage from data loss into delayed delivery. The design detail that matters is the ring buffer: when storage fills, something has to be discarded, and doing that by age is usually right while doing it by importance is usually better. Dropping the oldest routine samples before the oldest anomalies costs nothing to implement and preserves exactly the records someone will want.

The test worth applying to any telemetry design is simple and rarely applied. Take a real question the deployment exists to answer, and trace backwards through what is transmitted to see whether the answer is actually derivable. Teams frequently discover that they are shipping enormous volumes of data that cannot answer their own question, while the reading that would answer it was averaged away at the edge to save bandwidth.