/pylibup

Personal Pylib Builder

Primary LanguagePythonMIT LicenseMIT

pylibup

Personal and Opinionated Pylib Builder with CLI

Motivation

Since I've been on a recent building spree, I've been trying to adopt the DRY methodology. Mostly releasing smaller modules that aren't mono-repos. But copying and pasting templates got boring. So this sets to solve a few things that I use in particular, and likely won't be useful if your use case doesn't fit as well.

The end goal is that the structure should almost be pip install-able as is.

Tasks:

  • Setup a sane setup.py

  • Create some base module files in the correct structure

  • Adds .gitignores by default to prevent my default working files to be added

  • Creates CI/CD github workflows for:

    • Autopublish to Pypi

    • Docker Image releases (if part of an app)

  • Sets up repo secrets for dependent CI/CD workflows

Why should you use this over other ones such as cookie-cutter? I dont know, you probably shouldn't.

Quickstart

pip install --upgrade pylibup

Usage

cd ~/path/to/github
mkdir newlib && cd newlib

## This will generate a metadata.yaml in the cwd. You can then edit that file.

pylibup repo init

## Options & Args
# - name: Optional[str] = Argument(None) = name of python library
# - project_dir: Optional[str] = Argument(get_cwd()) = where the project should go
# - repo_user: Optional[str] = Argument(None) = your github username, or optionally, an org repo
# - github_token: Optional[str] = Option("", envvar="GITHUB_TOKEN") = auth token for github
# - private: bool = Option(True, "--public") = whether the repo should be published publicly, private by default.
# - overwrite: bool = Option(False) = overwrite existing files that were created
# - overwrite_state: bool = Option(False) = overwrite existing local state

## This will construct the library according to the metadata.yaml specs

pylibup repo build

## Options & Args
# config_file: Optional[str] = Argument(get_cwd('metadata.yaml')) = where your metadata.yaml should be located
# name: Optional[str] = Argument(None) 
# project_dir: Optional[str] = Argument(get_cwd()),
# github_token: Optional[str] = Option("", envvar="GITHUB_TOKEN"),
# pypirc_path: Optional[str] = Option("~/.pypirc", envvar="PYPIRC_PATH"),
# commit_msg: Optional[str] = Option("Initialize"),
# auto_publish: bool = Option(False), # If auto_publish == True, then will automatically push to github
# overwrite: bool = Option(False),
# overwrite_state: bool = Option(False),

## Used whenever you didnt specify auto_publish = True

pylibup repo publish

## Options & Args
# config_file: Optional[str] = Argument(get_cwd('metadata.yaml')),
# github_token: Optional[str] = Option("", envvar="GITHUB_TOKEN"), 
# pypirc_path: Optional[str] = Option("~/.pypirc", envvar="PYPIRC_PATH"),
# commit_msg: Optional[str] = Option("Initialize"),
# overwrite_state: bool = Option(False),

## Additionally you can utilize the build.sh script
sh build.sh dist # releases to main pypi
sh build.sh # will deploy to testpypi

Statefulness

CLI Apps tend to have bad statefulness. Two statefiles are created:

  • local: the current project folder

  • global: where the lib is installed

You can set certain configs like github_token to where it will always load, rather than having to set it each time.

The global state is loaded first, and overridden by local state values.

Note: if you reinstall this library, the global state will likely be erased.

pylibup state set github_token=ghp_xtyz anothervalue=1234

## Options and Args
# states: List[str] where a state is "x=y"
# --global-state: will write to global state versus local state
# --overwrite-state: will overwrite current values, true by default

"""
[pylib] app.set_state     Setting github_token -> ghp_xtyz. Previous: None
"""

Metadata Templating

Below is the base configuration for the metadata that is autogenerated

# These will automatically be added to .gitignore
gitignores:
- cache*
- '*.DS_Store'
- tests*
- __pycache__*
- '*logs'
- '*dist'
- '*build'
- '**build.sh'
- '*test.py'
- '*.egg-info*'
- '*.vscode'
- '**.ipynb'
- '**meta.yaml'
- '**metadata.yaml' # avoids adding this metadata file 
- '**state.yaml'
# Some optional configs
options:
  default_branch: main # sets to the default branch of the repo
  include_app: true # creates an app/ which is intended to use if containerizing
  include_buildscript: true # includes a build.sh, allowing you to quickly publish to pypi
  include_dockerfile: true # includes a Dockerfile [using fastapi]
  include_init: true # includes an __init__.py file in your module root that adds all the modules
  include_reqtext: true # includes a requirements.txt in the repo root
  private: true # sets the repo to public or private
project_description: '' # metadata used for description text
readme_text: '' #will be merged into the README.md
# optionally use annotation such as
# readme_text: |
#   ## My Readme
#   this is the readme 
repo: trisongz/pylibup # the full name of your repo 
secrets: # Optional secrets that will be automatically added to your repo.
  AWS_ACCESS_KEY_ID: # as this is a dict, the key AWS_ACCESS_KEY_ID will be set as the secret key
    from: AWS_ACCESS_KEY_ID_SVC_ACCOUNT # but `from` is the key that is used to get env value
  AWS_REGION: us-east-1 # as this is a string, it will use this value directly.
  AWS_SECRET_ACCESS_KEY: # as this is empty (null), will use the `key` to get the env value
  #  Additionally, you can set any other values following the same pattern as above.
  # if no value is found, then it will not attempt to set the value.
setup:
  author: null # will attempt to gather from github user profile
  cli_cmds: [] # cli cmds to be added where an item = `pylibup = pylibup.cli:baseCli`
  email: ts@growthengineai.com # will attempt to gather from github user profile
  git_repo: trisongz/pylibup 
  kwargs: {} # additional setup keyword dict values to add to main setup function
  lib_name: pylibup # if you intend for your `import x` to be different than the repo name
  pkg_name: pylibup # the intended repo name
  pkg_version: 0.0.0a # app version
  require_py3: true # forces a check prior to installation
  require_py3_version: 3.7 # no value = no check
  requirements: # individual library requirements
  - lazycls
  - pylogz
  - yourreq>1.5
structure: # will create these under yourapp/
  modules: # so in this example, the following are created
  - classes # yourapp/classes.py
  - client # yourapp/client.py
  - config # yourapp/config.py
  - utils # yourapp/utils.py
workflows: # automatically create github workflows
  docker_build: false # a quick docker-build.yaml targeting the Dockerfile when new pushes are made
  docker_build_options: # specific build options for docker-build.yaml
    # so the image handle will be img_repo/app_name
    app_name: '' # the image name you want to publish under
    docker_options: 
      img_repo: '' 
    ecr_options:
      img_repo: ''
    require_ecr: true # will create specific templating for ecr
  pypi_publish: true # will create a workflow for pypi publish on push of setup.py and releases. Will also attempt to set PYPI_API_TOKEN if pypi_path is found to enable automagic.

Libraries & Dependencies

  • typer: used for CLI

  • PyGithub: Used as Github API interface

  • GitPython: Used for git interface

  • Jinja2: Used for templating

  • pyyaml: Used for loading/saving statefiles

  • pylogz: Used for logging

  • requests: Used for calling the github API directly.