Install scrapy-save-to-qiniu using pip::
$ pip install scrapy-save-to-qiniu
- Add the
settings.py
of your Scrapy project like this:
QINIU_AK = ''
QINIU_SK = ''
QINIU_BUCKET = ''
QINIU_DOMAIN = ''
QINIU_DEL_SRC = True
QINIU_POLICY = {
}
QINIU_FIELDS = [
'pdf_url',
]
- Enable the pipeline by adding it to
ITEM_PIPELINES
in yoursettings.py
file and changing HttpCompressionMiddleware priority:
ITEM_PIPELINES = {
'scrapy_save_to_qiniu.pipelines.SaveToQiniuPipeline': -1,
}
The order should before your persist pipeline such as save to database and after your preprocess pipeline.
no need to change your code
Please use github issue
PRs are always welcomed.