Smoothing out spikes in data
Let us call the function $s(t)$. One approach we could do is to
Calculate local mean values $\mu(t)$ as $$\mu(t) = (l*s)(t)$$ Where $*$ is a convolution and $l$ is some function so that the convolution performs an estimate or approximation of local mean value: $$(l*s)(t)\approx\frac{1}{2\alpha}\int_{t-\alpha}^{t+\alpha}s(x)dx$$
Now consider $|\mu(t)-s(t)|$ and calculate the maximum over each $[-\alpha,\alpha]$ interval.
- Somehow sum up all of these maxima. This sum can be used as a measure to decide how much to smooth each point.
Which values of $\alpha$ we should use here will be dependent on the number of samples of each spike. We can also replace the maximum with a sort and locally distribute points over all the highest scorers.
This would become something of the opposite of a median filter in one sense. Instead of focusing on the median we focus on trying to quantify the outliers and how far out they lie.
Sorting can be a nasty operation but keep in mind that for this application we can use the redundancy of local neighbourhood overlaps to be able to do faster sorts.
Related videos on Youtube
Zishan Neno
Updated on August 01, 2022Comments
-
Zishan Neno over 1 year
I have a list of speed data for a driver (at infrequent intervals) that has certain spikes in it when the driver was too fast or suddenly braked. I'm looking for a formula that can smooth out the data so that my data is something similar to the chart below:
I've tried simple moving average and exponential moving average but is there any way of making the smoothed out data to stick/follow the trending data?
Might be worth mentioning that the data is real-time so the chart gets plotted on receiving each speed update.