Methods and defaults
How Sigmend checks and repairs sensor data
In short: Sigmend uses four fixed rules, one for each kind of problem. You can see and change every default. Gaps and spikes can be repaired with simple, standard methods after you review each change. Flatlines and drift are only flagged. The same file with the same settings always gives the same result.
How to detect gaps in time-series data
Gaps: missing time
Sigmend works out the usual time between readings, using the median interval of the series. When the time between two readings is more than N times that interval, it flags a gap. The default N is 2. So for a sensor that reports every minute, a step of more than two minutes is flagged.
Why the median: it ignores a few odd intervals, so one long outage does not change what counts as "normal".
Common causes: logger or network outages, historian compression settings, and time-zone or clock changes. In building-management trend data, gaps often follow controller restarts.
Default: N = 2
How to find spikes and outliers in sensor data
Spikes: single readings out of place
For each point, Sigmend looks at a rolling window around it (21 readings, centred on the point). It takes the median of that window and the median absolute deviation (MAD), which is the typical distance of points from that median. A point outside median ± k × MAD is flagged as a spike. The default k is 5. A lower k flags more points.
The median and MAD are used instead of the mean and standard deviation because a spike pulls the mean toward itself. The median does not move much.
Where a window is almost perfectly flat its MAD is close to zero, so Sigmend never lets the MAD fall below half of its typical value for that column. That keeps normal noise next to a flat run from being flagged.
You can also set a maximum step. Any change between two readings larger than that value is flagged, whatever the window shows.
Default: k = 5 · step Off
Stuck sensor detection
Flatlines: the same value, repeated
When a column holds exactly the same value for M readings or more in a row, Sigmend flags a flatline. The default M is 10.
Some signals really are flat, such as a closed valve or a setpoint. That is why flatlines are flagged and never changed automatically. Raise M for slow signals. Lower it for signals that should always move.
Default: M = 10
What is sensor drift?
Drift: a slow move away from the baseline
Drift is a slow, steady change in readings that does not come from the process. Ageing sensors and calibration loss are common causes. Day to day it is hard to see.
Sigmend compares a rolling mean with a baseline window at the start of the series. The baseline is the first 10% of the readings (at least 10, at most 500), and the rolling mean covers 5% of the series (at least 5, at most 500 readings). When the rolling mean moves more than the set percentage away from the baseline, the section is flagged. The default is 10%.
A real change in the process can look the same as drift. Treat a drift flag as a reason to check the sensor, not as proof that the sensor is wrong.
Default: 10% from baseline
How repairs work
- Linear interpolation (gaps). Missing readings are filled in on a straight line between the last reading before the gap and the first one after it. This suits short gaps in slow-changing signals. Over long gaps, the filled values are a guess and look smoother than real data.
- Median replacement (spikes). A flagged point is replaced with the rolling median of its window.
- Resampling to a fixed interval. The series is put on a regular time grid, for example one reading per minute. Many tools and models expect this.
Every change is shown before and after. You accept or reject each one. Accepted changes are listed in the change log, so anyone reading the repaired file can see what was changed.
What Sigmend never changes on its own
- Flatlines and drift. The right fix depends on the sensor and the process: a backup sensor, a calibration offset, or removing the section. Sigmend shows where they are and leaves the decision to you.
- Anything you rejected. Rejected changes leave the original data as it was.
- Your original file. Sigmend never edits the file you loaded. It writes a new file for you to download.
Glossary
- Time series
- Readings recorded with a timestamp, usually at a regular interval.
- Median interval
- The middle value of all the times between readings.
- MAD (median absolute deviation)
- The median distance of points from the median. It measures spread, and spikes barely affect it.
- Rolling window
- A fixed number of nearby points used to judge each reading.
- Interpolation
- Estimating missing values from the values around them.
- Resampling
- Moving readings onto a regular time grid.
- Baseline window
- The stretch of data used as "normal" when checking for drift.