felipensp/python-etl

ETL tool written in Python that provides an specific DSL which is translated to Python script to handle input data

python-etl

ETL tool written in Python that provides an specific DSL which is translated to Python script to handle input data.

DSL

/(pattern)/ {
    // Python code to run when pattern matches
    // We can use $1, $2, $N to refer to the matching group
}

or

<csv> {
    // Python code to run on every csv row
    // We can use $0, $1, $N to refer to csv column index
}

Examples

ETL file

result = []
/(?m:^)(\d{3})-([a-z]+)/ {
    result.append({"number": int($1), "description": 'good' if $2 == 'foo' else 'bad'})
}
/(\d+)/ {
    result.append({"number": int($1)})
}
print(result)

Using stdin in command-line

>python python-etl.py test.etl
123
^Z
[{'number': 123}]

>python python-etl.py test.etl
123-foo
^Z
[{'number': 123, 'description': 'good'}, {'number': 123}]

Using payload input file

>type test\payload | python python-etl.py test.etl
[{'number': 123, 'description': 'good'}, {'number': 1337}, {'number': 123}, {'number': 456}, {'number': 777}]

Processing CSV file

ETL file

<csv> {
   print($1)
}

Testing in command line

> python python-etl.py csv.etl
id,name
1,felipe
^Z
name
felipe