shypip (Secure Hybrid Pip) is a pip wrapper that improves security in hybrid public-private repository configurations.
Create a virtual environment and activate it:
$ python3 -m venv venv # or python3 -m virtualenv venv
$ source venv/bin/activate
Install the package from the github repository:
(venv) $ pip install git+https://github.com/mike10004/shypip.git
(venv) $ shypip --help # prints pip help text
TODO (not tested on Windows)
Usage is exactly the same as pip
. If the download
or install
command is
executed and a package dependency has installation candidates from multiple
repositories, the installation will fail instead of pip selecting the candidate
with highest version.
These environment variables are relevant to application behavior:
- SHYPIP_UNTRUSTED - comma delimited list of domains that are untrusted; default is
pypi.org
- SHYPIP_POPULARITY - minimum number of downloads to be eligible for installation; default is one million
- SHYPIP_CACHE - pypistats cache directory; default is under system temp directory
- SHYPIP_PYPISTATS_API_URL - pypistats API URL; default is
https://pypistats.org/api
- SHYPIP_MAX_CACHE_AGE - max age in minutes before pypistats cache files are considered stale; default is
1440
- SHYPIP_DUMP_CONFIG - if 1, print config to standard error and exit
- SHYPIP_PROMPT - canned answer to shypip install permission prompt
- SHYPIP_LOG_FILE - pathname of log file to append to
A package popularity query returns three values: downloads in the last day,
last week, and last month. If the popularity threshold (value of
SHYPIP_POPULARITY environment variable) is an integer, then all three values
must be exceed the threshold value for the threshold to be satisfied. More
granular control may be exercised by defining the threshold in URL query
string syntax, e.g. SHYPIP_POPULARITY='last_day=100&last_week=200'
. To allow
a threshold to be satisfied by any popularity value instead of all values, use
the prefix or:
, e.g. SHYPIP_POPULARITY='or:last_day=100&last_month=300'
.
To see how it works, you can set up a local repository that you consider "trusted" and
populate it with versions of packages that also exist in PyPI. (These instructions assume
you created a virtual environment that has shypip installed, as described in the
Installation section above.) Here we create a local repository that contains a
single package, sampleproject
version 1.3.0. The public repository has many other
versions, including version 1.3.1. We will demonstrate the use of shypip in the common
use case where a project declares a loose dependency sampleproject~=1.3.0
, which may
be satisfied by either version 1.3.0 or version 1.3.1.
(venv) $ mkdir -p demo/sampleproject
(venv) $ pip download --no-deps --dest demo/sampleproject sampleproject==1.3.0
(venv) $ python -m http.server 8080
That starts an HTTP server to serve your mini repository. Now open another terminal window to try out shypip:
$ source $VENV_DIR/bin/activate
$ export SHYPIP_LOG_FILE=/tmp/shypip.log
$ export SHYPIP_POPULARITY=100
(venv) $ shypip install 'sampleproject~=1.3.0' --extra-index-url http://localhost:8080/
Looking in indexes: https://pypi.org/simple, http://localhost:8080/
shypip: installation candidate sampleproject 1.3.1 from files.pythonhosted.org satisfies popularity threshold; allow (yes/no)? yes
Collecting sampleproject~=1.3.0
Downloading sampleproject-1.3.1-py3-none-any.whl (4.2 kB)
Collecting peppercorn
Downloading peppercorn-0.6-py3-none-any.whl (4.8 kB)
Installing collected packages: peppercorn, sampleproject
Successfully installed peppercorn-0.6 sampleproject-1.3.1
Installation pauses for user input because packages from both private (trusted) and public (untrusted) sources are available, and the package from the public repository satisfies the (low) popularity threshold of 100. If the user enters 'yes', the public package is installed fails, and if the user enters 'no', the private package is installed.
If any of the following conditions is true, the private package is installed without any prompt:
- packages from private and public sources are available, and the private package version is higher
- packages from private and public sources are available, but the public packages do not satisfy the popularity threshold
- relies on internal API of pip~=22.3.1, so compatibility is limited
- does not protect against typosquatting or bad dependency declaration; will allow installation of package from public (untrusted) repository if no package that satisfies a requirement is available from the private (trusted) repository
The code in this repository was produced as part of a project for a computer security course. Several people were involved in its design, implementation, and evaluation, and for their privacy their names are not included in the package metadata file, but their contributions are appreciated.