pip installable package to support Serenata de Amor and Rosie development.
Serenata_toolbox is compatible with Python 3+
$ pip install -U serenata-toolbox
Copy config.ini.example as config.ini and edit it with your own credentials. If you don't plan to upload anything to S3 please don't bother about keys and secrets in this file.
We have plenty of them ready for you to download from our servers. And this toolbox helps you get them. Let's say you want your datasets at data/:
from serenata_toolbox.datasets import Datasets
datasets = Datasets('data/')
# now lets see what are the latest datasets available
for dataset in datasets.downloader.LATEST:
print(dataset) # and you'll see a long list of datasets!
# and let's download one of them
datasets.downloader.download('2016-12-06-reibursements.xz') # yay, you've just downloaded this dataset to data/
# you can also get the most recent version of all datasets:
latest = list(dataset.downloader.LATEST)
datasets.downloader.download(latest)If the last example doesn't look that simple, there are some fancy shortcuts available:
from serenata_toolbox.datasets import fetch, fetch_latest_backup
fetch('2016-12-06-reibursements.xz', 'data/')
fetch_latest_backup( 'data/') # yep, we've just did exactly the same thingIf you ever wonder how did we generated these datasets, this toolbox can help you too (at least with the more used ones — the other ones are generated in our main repo):
from serenata_toolbox.federal_senate.dataset import Dataset
from serenata_toolbox.chamber_of_deputies.dataset import Dataset
senate = Dataset('data/')
senate.fetch()
senate.translate()
senate.clean()
chamber = Dataset('data/')
chamber.fetch()
chamber.translate()
chamber.clean()The full documentation is still a work in progress. If you wanna give us a hand you will need Sphinx:
$ cd docs $ make clean;make rst;rm source/modules.rst;make html
Within your virtualenv:
$ git clone https://github.com/datasciencebr/serenata-toolbox.git $ python setup.py develop
Always add tests to your contribution — if you want to test it locally before opening the PR:
$ python -m unittest discover tests
When the tests are passing, also check for coverage of the modules you edited or added — if you want to check it before opening the PR:
$ pip install coverage $ coverage run -m unittest discover tests $ coverage html $ open htmlcov/index.html
Follow PEP8 and best practices implemented by Landscape in the veryhigh strictness level — if you want to check them locally before opening the PR:
$ pip install prospector $ prospector -s veryhigh serenata_toolbox
If this report includes issues related to import section of your files, isort can help you:
$ pip install isort $ isort **/*.py --diff
Always suggest a version bump. We use Semantic Versioning – or in Elm community words:
- MICRO: the API is the same, no risk of breaking code
- MINOR: values have been added, existing values are unchanged
- MAJOR: existing values have been changed or removed
And finally take The Zen of Python into account:
$ python -m this