kevenli/scrapy-firebase

scrapy-firebase

Firebase pipeline for Scrapy.

Installation

Install via pip:

pip install scrapy-firebase

Configuration

Basic configuration example

Follow the steps on this guide, once you downloaded the json with firebase secrets, convert it to base64. On macos you can use the follwing command line to convert and copy to your clipboard:

cat firebase_secrets.json | openssl base64 | pbcopy

Add scrapy-firebase to your projects settings.py file and setup some variables.

ITEM_PIPELINES = [
  'scrapy_firebase.FirebasePipeline',
]

FIREBASE_SECRETS = """
  YOUR BASE64 ENCODED JSON HERE
"""

# Replace project-id to yours.
FIREBASE_DATABASE = 'https://project-id.firebaseio.com/'

# Insert an appropriate value.
FIREBASE_REF = ''

# To compose more robust child paths, you can add a list of properties.
FIREBASE_KEYS = ['uid', 'spider_name']