Is Numenta accepting PR and new algorithm submissions to NAB?
markNZed opened this issue · 5 comments
I have an anomaly detector that I would like to submit, is Numenta still accepting new submissions?
Hi,
Yes, we are. The submitter will need to sign the Contributor's License and meet these requirements.
Thanks!
Hi, it makes sense that changes to NAB need a contributors license. It is less clear about the actual detector, it seems strange to grant Numenta commercial rights to the intellectual property of all the detectors submitted. I guess Numenta does not have those rights for several of the detectors included in the scoreboard? Is there a way to submit a new detector with a typical open source license rather than transfer the intellectual property to Numenta? I guess the detector could be in another git repo and only the interface for running that detector included in NAB?
On a related topic I added an issue for an error in the window labels for one of the timeseries. How do we handle bugs like that - does a new detector run with the window corrected or does it run with the incorrect window (like the other detectors have been run)?
Hi, signing a contributor's license is pretty standard practice with open source projects. Ours is actually modeled after the Apache license. Basically in order for us to re-release any code, we need full rights from the contributor to do so. The contributor of course still retains their rights to do what they want. All code checked into NAB has gone through this license.
You can also submit a detector that's residing in another repository. One example is the Twitter detector. Only the interface for running it is in NAB. We'd want clear instructions for installation and running it. We do however require that it's open source, not for example, hidden behind some cloud API.
On a related topic I added an issue for an error in the window labels for one of the timeseries. How do we handle bugs like that - does a new detector run with the window corrected or does it run with the incorrect window (like the other detectors have been run)?
Is that #385? Unfortunately anything that might change the results requires another version of NAB so that results are exactly comparable in papers. NAB is in "maintenance mode" right now, so we can't really create a new version, no matter how minor, at the current time.
Is that #385?
Yes. There should be a bonus for detectors that find bugs in NAB :) I understand the rationale of keeping the incorrect window. But this penalizes detectors that detect early: instead of +1 for detecting it becomes -1 for not detecting, so quite a big difference.
I think there is a case to be made that this bug does not impact the results of the detectors and only influences how the score is calculated. So just rerunning the scoring would be sufficient?