niklasb/dryscrape

multiprocessing with dryscrape

Closed this issue · 0 comments

This is not an issue , but sharing how to deal with xvfb .

I am using celery to run multiple sessions of webkit-server . I need to remove ' dryscrape.start_xvfb() ' in the code and put xvfb-run before celery command , like this

xvfb-run -a celery -A googlescrape worker --loglevel=INFO --concurrency=10 -Q googlescrape_task

I spent couple of hours to figure out and hope can help somebody to save the time .