Develop / employ appropriate model error (residual) handling approach
vc1492a opened this issue · 6 comments
The central component of this approach to anomaly detection is the manner in which model residuals (errors) are analyzed / handled for classifying which time periods may be considered anomalous. Once our residuals have patterns we can expect / work with,
There's a lot of work out there on this topic, and it's something my colleagues and I explored in the past, too (non-parametric dynamic thresholding). But many approaches assume Gaussian distributions and certain patterns in the errors (dynamic thresholding, as an example, would not work well when residuals are high on the front and tail of the predicted values).
I've started playing around with an approach that utilizes the absolute errors and an examination of the previous N observations with a simple strategy of doing the following for each observation in the absolute errors:
1: Calculate the euclidean distance from this absolute error value to the previous N values, and collect those distances into an array.
2: For each of those distances, take the log transform.
3: Take the minimum distance value in the array as the distance value for the observation.
The idea borrows off of a previous approach my colleague Ian Colwell was using for anomaly detection for the Mars Science Laboratory (MSL), and aims to highlight localized anomalies over global ones. My desire for this sort of approach was that localized anomalies are what we are interested in here, and not larger, more global shifts in the telemetry. This sort of approach also accounts for model drift to some degree.
It remains to be seen whether this sort of approach will be ideal for this use case, but here's the initial results. The first image shows the approach's values for the day prior to the earthquake (a normal day) and the second image for the day of the earthquake. This approach seems to pick up on the anomaly, but there's still work to do in determining how to set a threshold for classification and applying it across different satellites etc.
@hamlinliu17 just a note that once you merge #51 and create a branch specific to this issue and handling model errors, if you could write the residual values to disk, that would allow anyone to work on the same set of errors. We could (and ought) to generate a set of errors for each satellite.
Of course, make sure first that you are able to perform the modeling appropriately and get a result similar to the above. In my latest modeling runs, I found that a small model with only the elevation as the input feature seems to work the most consistently and allows for rapid training times (due to the small model size).
@hamlinliu17 @MichelaRavanelli think I settled on a strategy which works for us, see the below screenshot.
The approach utilizes a modification of nonparametric dynamic thresholding which is from some of my earlier work with colleagues at JPL for spacecraft anomaly detection. I'm calling the above nonparametric consecutive assignment for now, but we can think about that and whether that's a fitting name.
For thoroughness, I have also coded a gaussian parametric approach which assumes the errors are normally distributed (gaussian). Our tests show this is not the case for our errors.
TODO:
- Encode the regular
nonparametric dynamic thresholdingapproach for comparison and thoroughness in the paper. I'd like to have the results from all three error handling approaches when we encode the metrics. - Have the code written for thresholding reviewed by a second pair of eyes (e.g., read the implementations of the thresholding approaches previously defined in literature and ensure they are correctly implemented).
Going to table this issue for now as we pivot to an image based modeling approach.
Since we are using an image based approach, this is no longer needed. 😄