How Long to Keep Industrial Data, and at What Resolution
Retention is usually decided by whatever the historian defaulted to, which is the wrong way round. What the questions you will ask later actually require, and where downsampling destroys the answer.
Almost every plant keeps its process data for however long the historian was configured to keep it when it was installed, which is to say the decision was made by a vendor default and a disk size. It is worth making deliberately, because the two failure modes are expensive in opposite directions: keeping everything at full resolution forever costs money and makes the data slow to query, while discarding the wrong thing means a question you will be asked in two years has no answer at all.
The useful way in is to work backwards from the questions. There are broadly four, and they want different things.
Operational questions cover the last few days: why did the line stop at 03:40, what was the pressure doing before the trip, which valve moved first. These need full resolution — the original sample rate, unsmoothed — because the whole point is sequence and detail. They also need to be fast, because somebody is asking while standing up. A few days to a few weeks of raw data is normal, and it is the smallest and most expensive tier.
Engineering questions cover months to a couple of years: is this pump degrading, has the yield on line two really dropped since the changeover, what does this machine's duty cycle look like across seasons. These tolerate downsampling but are ruined by the wrong kind. Averaging a signal to one point per minute destroys exactly the information a vibration or current trace carries; keeping minimum, maximum, average and sample count per interval preserves far more of it for a small extra cost. Which statistics you keep matters more than the interval you keep them at.
Compliance questions are set by somebody else and are not negotiable. Pharmaceutical, food, medical device, emissions and safety-related data have retention periods written into regulation or into a customer's quality agreement, and those periods are frequently longer than anyone's instinct — years past the life of the batch, sometimes past the life of the equipment. This tier also has requirements the others do not: the record must be attributable, unalterable and readable for the whole period, which is a statement about format and about audit trail, not only about storage.
Model questions are the newest tier and the one people underestimate. Training or retraining a model needs history, at a resolution that matches what the model consumes, with the labels attached. A plant that downsampled aggressively for three years and now wants to build an anomaly detector discovers it has averaged away the phenomenon it wants to detect. If there is any prospect of analytics on a signal, that signal's retention decision should be made with that in mind, because the cost of keeping it is small and the cost of not having it is a year of waiting.
Then a few practical points that recur.
Store raw and derived separately, and never overwrite raw with a corrected version. Corrections belong alongside the original with a reason attached; a value that was silently fixed is a value nobody can defend later.
Compression settings on a historian deserve a look rather than a default. Deadband and swinging-door compression discard points that fall within a tolerance, which is efficient and is also a decision about what detail no longer exists. A tolerance set generously enough to halve storage can flatten precisely the small excursions that a diagnosis depends on.
Time is the thing that makes cross-system questions answerable, and it is usually the weakest part. Data from a PLC, a historian, a MES and a lab system will be compared eventually, and if their clocks disagree by seconds the comparison is guesswork. Synchronised time and a stored, unambiguous timezone are cheap at design time and impossible to retrofit into old records.
Finally, write the retention policy down as a table: signal group, raw retention, aggregate retention, what statistics, where it lives, and why. It takes an afternoon, it makes the storage bill explainable, and it means the answer to "do we still have that data?" is something you can look up rather than something you have to go and find out.