niqdev/packtpub-crawler

Error when getting config file

trancen opened this issue · 3 comments

trancen commented

Updated to the latest version.

I have the following in a crontab, been using this for a few weeks working correctly. I noticed that I wasn't getting any new books so I ran the commend manually :

python /home/david/packtpub-crawler/script/spider.py --config /home/david/packtpub-crawler/config/prod.cfg -t pdf --extras

But I'm getting the following error.

[*] 2017-01-31 08:34 - fetching today's eBooks
[-] <type 'exceptions.IOError'> file not found! | spider.py@89
Traceback (most recent call last):
  File "/home/david/packtpub-crawler/script/spider.py", line 89, in main
    config = config_file(dir_path + args.config)
  File "/home/david/packtpub-crawler/script/utils.py", line 24, in config_file
    raise IOError('file not found!')
IOError: file not found!
[*] done

But if I move to the packtpub-crawler directory it works:

david@server:~/packtpub-crawler$ python script/spider.py --config config/prod.cfg -t pdf --extras


[*] 2017-01-31 08:39 - fetching today's eBooks
[*] configuration file: /home/david/packtpub-crawler/config/prod.cfg
[*] getting daily free eBook
[*] fetching url... 200 | https://www.packtpub.com/packt/offers/free-learning
[*] fetching url... 200 | https://www.packtpub.com/packt/offers/free-learning
[*] fetching url... 200 | https://www.packtpub.com/account/my-ebooks
[+] book successfully claimed
juzim commented

There were some issues with relative paths in cron: ce03d4f

Please try using relative paths (--config config/prod.cfg). You can also look at the readme file for working examples.

niqdev commented

@trancen using relatives path solves your problem? Thanks

trancen commented

Ya it did.. I had to use the following on my crontab

cd /home/david/packtpub-crawler/ && python script/spider.py --config config/prod.cfg -t pdf --extras

  • 👍1