Repository for How to set up a Python Repo for Automation and Collaboration.
# Clone this repo and change directory
git clone git@github.com:eugeneyan/python-collab-template.git
cd python-collab-template
# Install [Rye](https://rye-up.com/guide/) if it's not available on your system
make install-rye
# Sync Rye to your project specifications (e.g. .python-verision, added dependencies)
# Run the suite of tests and checks
make check
Make a pull request to this repo to see the checks in action 😎
Here's a sample pull request which initially failed ❌ the checks, and then passed ✅.
In it, we cover the following aspects of setting up a python project, including:
@pytest.fixture
def lowercased_df():
string_col = ['futrelle, mme. jacques heath (lily may peel)',
'backstrom, major. karl alfred (maria mathilda gustafsson)']
df_dict = {'string': string_col}
df = pd.DataFrame(df_dict)
return df
def test_extract_title(lowercased_df):
result = extract_title(lowercased_df, col='string')
assert result['title'].tolist() == ['mme', 'ms', 'mr', 'lady', 'major']
def test_extract_title_with_replacement(lowercased_df):
title_replacement = {'mme': 'mrs', 'ms': 'miss', 'lady': 'rare', 'major': 'rare'}
result = extract_title(lowercased_df, col='string', replace_dict=title_replacement)
assert result['title'].tolist() == ['mrs', 'miss', 'mr', 'rare', 'rare']$ rye run pytest
============================= test session starts ==============================
platform darwin -- Python 3.8.2, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
rootdir: /Users/eugene/projects/python-collaboration-template/tests/data_prep
collected 2 items
test_categorical.py::test_extract_title PASSED [ 50%]
test_categorical.py::test_extract_title_with_replacement PASSED [100%]
============================== 2 passed in 0.30s ===============================$ rye run pytest --cov=src
============================================================================================================= test session starts ==============================================================================================================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /Users/alex.furrier/git_repositories/python-collab-template
configfile: pyproject.toml
plugins: cov-5.0.0
collected 8 items
tests/data_prep/test_categorical.py ... [ 37%]
tests/data_prep/test_continuous.py ..... [100%]
---------- coverage: platform darwin, python 3.12.2-final-0 ----------
Name Stmts Miss Cover
--------------------------------------------------
src/__init__.py 0 0 100%
src/data_prep/__init__.py 0 0 100%
src/data_prep/categorical.py 12 0 100%
src/data_prep/continuous.py 11 0 100%
--------------------------------------------------
TOTAL 23 0 100%
======================================= 8 passed in 0.75s ========================================================
$ rye lint src/data_prep/categorical -v
$ rye run mypy src
src/data_prep/categorical.py:34: error: Incompatible default for argument "replace_dict" (default has type "None", argument has type "dict[Any, Any]") [assignment]
src/data_prep/categorical.py:34: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
src/data_prep/categorical.py:34: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
Found 1 error in 1 file (checked 4 source files)
clean-pyc:
find . -name '*.pyc' -exec rm -f {} +
find . -name '*.pyo' -exec rm -f {} +
find . -name '*~' -exec rm -f {} +
find . -name '__pycache__' -exec rm -fr {} +
clean-test:
rm -f .coverage
rm -f .coverage.*
clean: clean-pyc clean-test
test: clean
. .venv/bin/activate && py.test tests --cov=src --cov-report=term-missing --cov-fail-under 95
# .github/workflows/tests.yml
name: Tests
on: push
jobs:
tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v1
with:
python-version: 3.8
architecture: x64
- run: make setup
- run: make check
- run: bash <(curl -s https://codecov.io/bash)
Docker is often used for environment management and deployment of production code.
This repo is setup to package things in a Docker image for this purpose.
Through the use of Docker Compose a dev environment can also be stood up and torn down quickly. Docker compose allows for better environment setup through connected services (e.g. databases, etc) for closer replication of a production environment.
The docker compose file docker/docker-compose.yml builds an image from docker/Dockerfile and runs a bash shell.
Environment variables can be added in the relevant section of the docker-compose.yml if they are provided in a .env file within the docker directory. By default the .env file is excluded from the repo since it may contain secrets. Instead the file docker/template.env is provided which should provide non secret environment variables and the variable name for required secrets.
To create a dev environment run:
make dev-envThis should create a running docker container with everything required for development in this repo.
All other Make commands should still work as before.
All changes made to relevant files inside the container will be reflected outside the container as they are bound in the volumes section of the docker-compose.yml file. Any newly added directories or files will need to be added to the docker/Dockerfile with a COPY command and bound as a volume in the docker compose file.
Once development is finished and the project is ready to be deployed it can be built and tagged as a Docker image with:
make build-imageThe image name and tag are set in the Makefile variables IMAGE_NAME and IMAGE_TAG.
If the name of the image is a container registry, the image can be pushed to the registry with:
make push-image👉 View the article for the walkthrough.
- Update requirements.txt to use
poetry