orangain/scrapy-s3pipeline

Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.

PythonMIT

Issues

Json object are not delimited and not within array using JsonLinesItemExporter
#9 opened 4 years ago by chipzzz
1
What if len(self.items) < self.max_chunk_size for a long time
#3 opened 4 years ago by pokliu
1
Add S3 Access Keys
#1 opened 6 years ago by leovigna
2