Let other users add and run their own crawlers on our platform
sunu opened this issue · 1 comments
sunu commented
One potential solution for this is to have a webhook that listens for changes on a fixed set of git repositories and pull crawlers from them into our repository of crawlers.
Note to self:
Possible ways to do it:
- Webhook triggered
- Pull github repo
- Push them into the central repo of crawlers
Option A
- Figure out what changed(how?)
- Flush the remaining ops for the crawlers that changed from the queue
- Reload the crawlers into the manager
Option B
- Let the user explicitly tell us to reload the crawler
- Flush the ops
- Reload the crawler
sunu commented
Since we got rid of the UI and memorious is now more of a CLI tool and not a platform, this issue isn't valid anymore.