Smoothing out spikes in data

1,314

Let us call the function $s(t)$. One approach we could do is to

  1. Calculate local mean values $\mu(t)$ as $$\mu(t) = (l*s)(t)$$ Where $*$ is a convolution and $l$ is some function so that the convolution performs an estimate or approximation of local mean value: $$(l*s)(t)\approx\frac{1}{2\alpha}\int_{t-\alpha}^{t+\alpha}s(x)dx$$

  2. Now consider $|\mu(t)-s(t)|$ and calculate the maximum over each $[-\alpha,\alpha]$ interval.

  3. Somehow sum up all of these maxima. This sum can be used as a measure to decide how much to smooth each point.

Which values of $\alpha$ we should use here will be dependent on the number of samples of each spike. We can also replace the maximum with a sort and locally distribute points over all the highest scorers.

This would become something of the opposite of a median filter in one sense. Instead of focusing on the median we focus on trying to quantify the outliers and how far out they lie.

Sorting can be a nasty operation but keep in mind that for this application we can use the redundancy of local neighbourhood overlaps to be able to do faster sorts.

Share:
1,314

Related videos on Youtube

Zishan Neno
Author by

Zishan Neno

Updated on August 01, 2022

Comments

  • Zishan Neno
    Zishan Neno over 1 year

    I have a list of speed data for a driver (at infrequent intervals) that has certain spikes in it when the driver was too fast or suddenly braked. I'm looking for a formula that can smooth out the data so that my data is something similar to the chart below:

    Smoothed out chart data

    I've tried simple moving average and exponential moving average but is there any way of making the smoothed out data to stick/follow the trending data?

    Might be worth mentioning that the data is real-time so the chart gets plotted on receiving each speed update.