python_template

Summary

Oftentimes the initial setup of a Python repo can take a few minutes to a couple hours. By laying the foundation to rapidly implement an idea, can focus on the good bits instead of devops drudgery.

Caveat Emptor

Very little of this gets tested on Windows hosts. Windows Subsystem for Linux (WSL) is used where necessary with the default Ubuntu LTS install.

Be the change et al if Windows is your main and you wanna raise a PR with broad instructions on getting tooling working under Windows (e.g., docker, poetry, playwright.)

Table of Contents

python_template

Setup

Install
- editorconfig
- wsl
- asdf
- poetry
- docker-compose
- playwright

Usage

Mac and Linux users

Development environments and tooling are first-class citizens on macOS and *nix. For Windows faithfuls, please setup WSL below.

Windows Subsytem for Linux (wsl)

WSL allows Windows users to run Linux (Unix) locally at a system-level. All of the standard tooling is used and community guides can be followed without standard Windows caveats (e.g., escaping file paths, GNU utilities missing, etc.)

Install from the Setup section
Enable
- Start Menu > search for "Turn Windows Features On" > open > toggle "Windows Subsystem for Linux"
- Restart
M1 Macs only (Intel Macs and native Windows boxes need not apply)
- Revert WSL 2 to WSL 1 due to nested virtualization not being available at a hardware level
```
wsl --set-default-version 1
```
- Docker won't run without paravirtualization enabled, but the rest of the development environment will work as expected

Install Ubuntu

# enable default distribution (Ubuntu)
wsl --install ubuntu

Start Linux and prep for environment setup

# launch Ubuntu
ubuntu

# upgrade packages (as root: `sudo -s`)
apt update && apt upgrade -y

# install the community python repo (PPA)


# create standard user
adduser <username>
visudo

# search for 'Allow root to run any commands anywhere', then append identical line with new user
root            ALL=(ALL)       ALL
<username>      ALL=(ALL)       ALL

# Allow members of group sudo to execute any command
%sudo  ALL=(ALL) NOPASSWD: ALL

Additional configuration options

Configuration locations

WSL 1: /etc/wsl.conf

WSL 2: ~/.wslconfig

  # set default user
  [user]
  default=<username>

  # mounts host drive at /mnt/c/
  [automount]
  enabled = true
  options = "uid=1000,gid=1000"

  # WSL2-specific options
  [wsl2]
  memory = 8GB   # Limits VM memory in WSL 2
  processors = 6 # Makes the WSL 2 VM use two virtual processors

After making changes to the configuration file, WSL needs to be shutdown for 8 seconds
- wsl --shutdown

OPTIONAL: Change home directory to host Windows' home

# copy dotfiles to host home directory
cp $HOME/.* /mnt/c/Users/<username>

# edit /etc/passwd
<username>:x:1000:1000:,,,:/mnt/c/Users/<username>:/bin/bash

asdf

Install from the Setup section

WSL/Ubuntu Linux dependencies

sudo apt install \
make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget \
curl llvm libncursesw5-dev xz-utils tk-dev \
libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

All operating systems

# add python plugin
asdf plugin-add python

# install stable python
asdf install python latest

# uninstall version
asdf uninstall python 3.9.6

# refresh symlinks for installed python runtimes
asdf reshim python

# set stable to system python
asdf global python latest

# optional: local python (e.g., python 3.9.10)
cd $work_dir
asdf list-all python 3.9
asdf install python 3.9.10
asdf local python 3.9.10

# verify python shim in use
asdf current

# check installed python
asdf list python

Python pip

If a basic virtual environment (venv) and requirements.txt are all that's needed, can use built-in tools.

# create a virtual environment via python
python3 -m venv .venv

# activate virtual environment
source .venv/bin/activate

# install dependencies
python3 -m pip install requests inquirer

# generate requirements.txt
python3 -m pip freeze > requirements.txt

# exit virtual environment
deactivate

Poetry

NOTE: it's possible to use the built-in .venv virtual environment (e.g., troubleshooting SolverProblemError dependency hell)
```
poetry env use .venv/bin/python
```
Install from the Setup section

Normal usage

# Install (modifies $PATH)
curl -sSL https://install.python-poetry.org | $(which python3) - # append `--no-modify-path` to EOL if you know what you're doing 

# Change config
poetry config virtualenvs.in-project true           # .venv in `pwd`
poetry config experimental.new-installer false      # fixes JSONDecodeError on Python3.10

# Activate virtual environment (venv)
poetry shell

# Deactivate venv
exit  # ctrl-d

# Install multiple libraries
poetry add google-auth google-api-python-client

# Initialize existing project
poetry init

# Run script and exit environment
poetry run python your_script.py

# Install from requirements.txt
poetry add `cat requirements.txt`

# Update dependencies
poetry update

# Remove library
poetry remove google-auth

# Generate requirements.txt
poetry export -f requirements.txt --output requirements.txt --without-hashes

# Uninstall Poetry (e.g., troubleshooting)
POETRY_UNINSTALL=1 bash -c 'curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py' | $(which python3) -

Docker

Install from the Setup section

Usage

# clean build (remove `--no-cache` for speed)
docker-compose build --no-cache --parallel

# start container
docker-compose up --remove-orphans -d

# exec into container
docker attach hello

# run command inside container
python hello.py

# destroy container
docker-compose down

Docker Troubleshooting

Watch logs in real-time: docker-compose logs -tf --tail="50" hello

Check exit code

$ docker-compose ps
Name                          Command               State    Ports
------------------------------------------------------------------------------
docker_python      python manage.py runserver ...   Exit 0

Playwright

Install from the Setup section

Usage

# install
pip install --upgrade pip
pip install playwright
playwright install

# download new browsers (chromedriver, gecko)
npx playwright install

# generate code via macro
playwright codegen wikipedia.org

Django

Follow the official Django Docker Compose article
- cd django
- Generate the server boilerplate code
```
docker-compose run web django-admin startproject composeexample .
```
- Fix upstream import bug and whitelist all hosts/localhost
```
$ vim composeexample/settings.py
import os
...
ALLOWED_HOSTS = ["*"]
```
- Profit
```
docker-compose up
```
- Optional: Comment out Django exclusions for future commits
  - Assumed if extracting Django boilerplate from template and creating a new repo
```
# .gitignore
# ETC
# django/composeexample/
# django/data/
# django/manage.py
```

GitHub Actions

Update submodules recursively

Add the submodule to the downstream repo

git submodule add https://github.com/pythoninthegrass/automate_boring_stuff.git
git commit -m "automate_boring_stuff submodule"
git push

Create a personal access token called PRIVATE_TOKEN_GITHUB with repo permissions on the downstream repo
- repo:status
- repo_deployment
- public_repo
Add that key to the original repo
- Settings > Security > Secrets > Actions
- New repository secret

Setup a new Action workflow

Actions > New Workflow
Choose a workflow > set up a workflow yourself

# main.yml
# SOURCE: https://stackoverflow.com/a/68213855
name: Send submodule updates to parent repo

on:
push:
    branches: 
    - main
    - master

jobs:
update:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
        with: 
        repository: username/repo_name
        token: ${{ secrets.PRIVATE_TOKEN_GITHUB }}

    - name: Pull & update submodules recursively
        run: |
        git submodule update --init --recursive
        git submodule update --recursive --remote
    - name: Commit
        run: |
        git config user.email "actions@github.com"
        git config user.name "GitHub Actions - update submodules"
        git add --all
        git commit -m "Update submodules" || echo "No changes to commit"
        git push

Debugging

asdf

No version set for command python

Make sure python or python3 isn't aliased in ~/.bashrc or ~/.zshrc

bash - Is it possible to check where an alias was defined? - Unix & Linux Stack Exchange

PATH

asdf, poetry, and python all need to be sourced in your shell $PATH in a specific order
- asdf stores its Python shims in ~/.asdf/shims
- poetry lives in ~/.local/bin
```
export ASDF_DIR="$HOME/.asdf"
export PATH="$ASDF_DIR/shims:$HOME/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"
```

Furthermore, any aliases or alias files need to be sourced as well

. "$ASDF_DIR/asdf.sh"
. "$ASDF_DIR/completions/asdf.bash"
. /usr/local/etc/profile.d/poetry.bash-completion

TODO

~~Add boilerplate to hello.py~~
~~Poetry~~
~~Dockerfile~~
~~Playwright~~
~~Django~~
- Merge with docker_python and put the latter on an ice float
- ~~Break out into separate folder~~
Flask
- Bonus points for Svelte front-end ❤️
- Break out into separate folder
~~asdf~~
terraform
k8s
wsl
- ~~enable~~
- ~~.wslconfig options~~
- ~~install ppa:deadsnakes/ppa~~
  - precluded by asdf (née pyenv)
- VSCode
  - Remote WSL install and usage
    - Or at least further reading nods
Debugging
- ~~$PATH~~
- Dependencies
- script itself via icecream

pythoninthegrass/knockknock_poc

python_template

Summary

Caveat Emptor

Setup

Usage

Mac and Linux users

Windows Subsytem for Linux (wsl)

asdf

Python pip

Poetry

Docker

Docker Troubleshooting

Playwright

Django

GitHub Actions

Update submodules recursively

Debugging

asdf

PATH

TODO

Further Reading