Can anomalies from one detector be applied before series is passed to next detector/assembly?
Opened this issue · 3 comments
The existing examples all seem to envision serial transforms feeding into a single detector which are then logically connected.
I don't see any where a simple detector is used for a preliminary detection, values are screened and the most egregious anomalies deleted, then the next detector is used. Is this possible? It is a common pattern of work. For example, a Seasonal decomposition is more robust if ridiculous values (ThresholdAD) have been removed (I think the existing ThresholdAD is broken for missing data, but making it work is only a one line fix).
Hi @water-e
From one user to another. You can do that using the adtk Pipeline
Pipe
object and using aCustomizedTransformer1D
see https://adtk.readthedocs.io/en/stable/notebooks/demo.html#CustomizedTransformer and https://adtk.readthedocs.io/en/stable/notebooks/demo.html#Pipeline it is quite flexible and you can add whatever steps you like with CustomizedTransformer1D
and CustomizedDetector1D
.
The Pipe
and Pipeline
methods take a bit of work to get familiar and if it proves too difficult or complicated, there is no reason why you cannot just transform the series and anomalies output in your own code with your own subsequent function/s and then pass the result to the next adtk method you wish to use.
As for the existing ThresholdAD
being broken for missing data I cannot say. I have not used ThresholdAD
on anything really. However, if you have debugged it and fixed it may be useful to open another issue describing it fully and adding an example data set and the code snippet that can be used to reproduce the issue and the code snippet to fix the issue. There has not been any maintenance activity on this repo for some time, so you just documenting the issue may help someone that runs into it in the future.
Thanks for the quick response. I misspoke about TransformAD. I meant the SeasonalAD has issues with NaN
If this can be done with a pipeline, I'd still like to hear a sketch of how.