/sling-python

Python wrapper for the Sling CLI tool

Primary LanguagePythonMIT LicenseMIT

logo

Slings from a data source to a data target.

Installation

pip install sling

Then you should be able to run sling --help from command line.

Running a Extract-Load Task

CLI

sling run --src-conn MY_PG --src-stream myschema.mytable \
  --tgt-conn YOUR_SNOWFLAKE --tgt-object yourschema.yourtable \
  --mode full-refresh

Or passing a yaml/json string or file

sling run -c '
source:
  conn: MY_PG
  stream: myschema.mytable

target:
  conn: YOUR_SNOWFLAKE
  object: yourschema.yourtable

mode: full-refresh
'
# OR
sling run -c /path/to/config.json

From Lib

from sling import Sling

config = {
  'source': {
    'conn': 'MY_PG',
    'stream': "select * from my_table",
  },
  'target': {
    'conn':  "s3://my_bucket/my_folder/new_file.csv",
  },
}

Sling(**config).run()

Config Schema

--src-conn/source.conn and --tgt-conn/target.conn can be a name or URL of a folder:

  • MY_PG (connection ref in db, profile or env)
  • postgresql://user:password!@host.loc:5432/database
  • s3://my_bucket/my_folder/file.csv
  • gs://my_google_bucket/my_folder/file.json
  • file:///tmp/my_folder/file.csv (local storage)

--src-stream/source.stream can be an object name to stream from:

  • TABLE1
  • SCHEMA1.TABLE2
  • OBJECT_NAME
  • select * from SCHEMA1.TABLE3
  • /path/to/file.sql (if source conn is DB)

--tgt-object/target.object can be an object name to write to:

  • TABLE1
  • SCHEMA1.TABLE2

Example as JSON

{
  "source": {
    "conn": "MY_PG_URL",
    "stream": "select * from my_table",
    "options": {}
  },
  "target": {
    "conn": "s3://my_bucket/my_folder/new_file.csv",
    "options": {
      "header": false
    }
  }
}