Advice on how to run Scrapy template without using main.py

Question

Advice on how to run Scrapy template without using main.py

Opened this issue 10 months ago · 2 comments

I'd like my project to have several CLI commands, e.g. using click. It's not entirely clear to me how to properly move the __main__.py logic somewhere else, so that I can wrap running the actor/spider into one command and have other commands doing other stuff.

Vaguely related to apify/apify-sdk-python#176

Answer 1 · 2024-01-17T10:35:40.000Z

I did some changes in my implementation so that it's possible:

Introduced CLI where only the logging setup is hoisted before everything else: https://github.com/juniorguru/plucker/blob/6fe0c31097b00339cbc05b2ab40fc1dae23160bd/juniorguru_plucker/cli.py
Moved all logging setup to a separate file: https://github.com/juniorguru/plucker/blob/main/juniorguru_plucker/loggers.py

Feel free to grab inspiration from what I did, or even chunks of code (MIT licensed, just mention my name). I guess I've solved this for myself now. If there are updates to the Scrapy template, I hope I'll be able to somehow keep up with it and backport changes to my highly customized project.

Answer 2 · 2024-01-19T07:46:02.000Z

Hi Honza, thank you for opening this. Moving as much code as we can from the template to the SDK is definitely a good way to go. Unfortunately, adding new features to our Scrapy-Apify integration is not a priority for this quarter, so I cannot promise I'll have time to take a look at this in the near future.