This is a set of scripts to get statistics on the scikit-surgery library and turn them into a nice webpage
We provide a repository template for creating a sustainability dashboard for your own library/ecosystem of interest. It automatically creates the scripts and files needed to run the analysis needed for deployment of dashboard showing metrics of libraries existing within a given base Python package/ecosystem. It includes the Github Action that deploys the produced html files to the gh-pages branch of a target repository, which triggers a deployment every 12 hours, using the cron scheduler.
- You can use your preferred virtual python envirohment to install cookieninja.
pip install cookieninja
For instance, you can create conda or mamba environment with basic packages as follows
conda create -n susdbVE pip cookieninja -c conda-forge conda activate susdbVE
- Run cookieninja in the desired location
cookieninja gh:scikit-surgery/sustainable-pkg-stats
If you have this repo locally (this may be the case if you are developing, or you cloned this repository before), you can alternatively run the following:
cookieninja /path/to/your/checkout/of/python-template
3. A series of questions will pop up to configure the project. Type the answer or hit return to use the default option (shown in square brackets). Note that these project variables are defined in the cookiecutter.json file. It is important that you enter a value for base_library_name as the dashboard analysis scripts will be configured for this base package. There is a script the cookieninja runs placed under hooks/pre_gen_project.py that checks if the name given returns package entries in pypi search.
author_name [John Smith]: author_email [temp@gmail.com]: project_name [Community Dashboard]: project_slug [dashboard_for_scikit-surgery]: base_library_name [scikit-surgery]: project_short_description [A dashboard template from scikit-surgery]: funder [JBFC: The Joe Bloggs Funding Council]: Select licence: 1 - MIT 2 - BSD-3 3 - GPL-3.0 Choose from 1, 2, 3 [1]:
4. This will create a directory with the following configuration: For example, for a project with the following variables:
project_name : Community Dashboard base_library_name : scikit-surgery
We will get a project folder named after dashboard_for_scikit-surgery, structured like this:
├── assets │ └── logo-dashboard.svg ├── _config.yml ├── get_badges.py ├── get_github_repos.py ├── get_loc.py ├── get_pypi_repos.py ├── html │ ├── dashboard.html │ ├── dashboard.html.in.head │ ├── dashboard.html.in.tail │ ├── excluded.html.in.head │ ├── excluded.html.in.tail │ └── exclusions.html ├── index.html ├── libraries │ ├── exclusions │ └── lines_of_code ├── LICENSE ├── loc │ ├── CMakeCatchTemplate.html │ └── PythonTemplate.html ├── pypi-simple-search ├── README.md ├── requirements.txt ├── sksurgerystats │ ├── common.py │ ├── from_github.py │ ├── from_pypi.py │ ├── html.py │ ├── __init__.py │ ├── __pycache__ │ │ ├── common.cpython-310.pyc │ │ ├── html.cpython-310.pyc │ │ └── __init__.cpython-310.pyc │ └── pypi_downloads.py ├── static │ └── loc_plot.js ├── templates │ ├── dashboard.css │ └── loc_plot.html ├── tests │ ├── conftest.py │ └── test_template_workflow.py ├── update_dashboard.py ├── update_github_stats.py └── update_pypi_stats.py
Important configurations to note:
- get_github_repos.py and get_pypi_repos.py will take base_library_name as the base name to search packages in https://pypi.org/search/ and github
- project_name will appear in the README.md as the human-readable name of the project.
- html/dashboard.html will take project_name as the main title, Community Dashboard, and also use project_slug for a description below the logo, as shown below:
- Create a new repository with the same project_slug name (e.g. https://github.com/$GITHUB_USER_ID/dashboard_for_scikit-surgery).
- Few [optional] things to set before you can run the pipeline!
You can specify a list for the libraries you want to exclude from your dashboard deployment, under libraries/exclusions
- Similar to libraries folder, this (as shown below) has a dict entry for each package, such as in this example from scikit-surgery:
- libraries/exclusions├── scikit-surgeryoverlay├── scikit-surgerytorsosimulator└── scikit-surgeryvideoutils
Each file entry (ex. scikit-surgeryoverlay) is a .json file that has : an obsolete key and a value that is a sentence describing why they are obsolete, such as:
`{"obsolete" : "Became <a href='https://github.com/UCL/scikit-surgeryvtk'>sikit-surgeryvtk.</a>"}`You can save the logo of your base package (a .svg file) under assets/logo-dashboard.svg for it to show up in your deployment header
7. Github Configurations a. You need to initialise github pages in your repository and set the deployment source from branch main: You can find the instructions here
You might need admin rights from your organisation to use your organisation's base name. You can also use your username as the domain.
Your configuration will need to look like this (In the example below, our domain name is the scikit-surgery organisation):
8. Setting up git repository. Push your first commit to your package repository
cd dashboard_for_scikit-surgery git init git add . git commit -m "first commit" git branch -M main git remote add origin git@github.com:$GITHUB_USER_ID/dashboard_for_scikit-surgery.git git push -u origin main
- Running the pipeline on Github actions
The Github Actions workflow will run this pipeline, so you do not need to do anything.
- Running the pipeline locally
Locally, you can check if the pipeline works correctly, by running the python scripts ordered and referenced in the Makefile file of this repository. Note for checking if things work properly. While running get_badges.py you should notice that under libraries folder, there are .json files of dictionary entries for each package.
To run the scripts locally, you need a personal access token for Github API generated from here Save it in the base directory in a text file named github.token Be careful not to commit the token file to your git repository. If you do it will be submitted to github in your publicly visible repository.
- (Optional) Generate your SSH keys as suggested here
- (Optional) GitHub CLI as suggested here
- Clone the repository by typing (or copying) the following line in a terminal at your selected path in your machine:
git clone git@github.com:SciKit-Surgery/sustainable-pkg-stats.git
Using conda
conda create -n susdbVE pip -c conda-forge conda activate susdbVE pip install -r requirements.txt
Using venv
mkdir env python -m venv env/ source env/bin/activate pip install -r requirements
Running the pipeline that generates dashboard.html and associated files needed by Github Pages
bash Makefile
You can also run the individual python scripts to check outputs:
Search for relevant packages on pypi and githib
python get_pypi_repos.py python get_github_repos.py
update stats
python update_pypi_stats.py python update_github_stats.py
get coverage/docs/etc badges
python get_badges.py
update html files
python update_dashboard.py
Inspect libraries with pypi
./pypi-simple-search scikit-surgery > scikit-surgery-onpypi.txt python get_github_repos.py > scikit-surgery-ongithub.txt
We can use pypinfo to get data for things on pypi
pypinfo --auth snappy-downloads-3d3fb7e245fd.json pypinfo scikit-surgeryvtk country