Design/Architecture: how to run some checks of recipe configuration from the UI (dry run)
Opened this issue · 2 comments
We regularly face situations where a recipe configuration is not correct and the UI cannot check it. Some simple checks are possible, but they cannot rely on advanced logic typically found in the scraper.
For instance, for a Youtube recipe if the editor enter a wrong playlist ID, he won't be notified about this until the task has found a worker, the worker has started the task and the scraper has failed. Or for a zimit recipe, the youzim.it UI does only basic checks about URL validity, but more advanced ones are only done in the scraper once the task has started.
How could we design a system which will allow to:
- make some scraper logic available in the UI to confirm recipe configuration is valid
- run this logic fast (i.e. not as a task which will have to wait for a worker to be available)
Suggestions welcomed!
What would be great is to have a draft list of real things we would like to check and must be run against upstream site infra,
Removed from zimit2 project unfortunately