build error: Package 'setuptools' requires a different Python: 2.7.12 not in '>=3.5'

I am trying to build the most recent version of master (9ef2fb9 as of now) of this repository by issuing make test, but I am facing a dependency error:

Requirement already satisfied: setuptools in ./local/lib/python2.7/site-packages (from pyramid==1.5.3->-r requirements.txt (line 3)) (45.0.0)
ERROR: Package 'setuptools' requires a different Python: 2.7.12 not in '>=3.5'
Makefile:27: recipe for target 'local/COMPLETE' failed
make: *** [local/COMPLETE] Error 1

It looks like this is a Python 3 setuptools as it wants Python 3 🤔. Which OS/distribution and how did you install Python (2.7)?

Thank you for your answer. It's Ubuntu 16.04.7 LTS and Python is installed via the regular distribution package repository. However, as far as I understand, this Python interpreter from the operating system is not being used as an own interpreter and all packages are installed into the ./local/ folder in an own virtual Python environment.

Yes it uses a virtual environment, I just wanted to compare the OS-level Python setup with ours (and treat the Makefile-internal venv setup as a blackbox), which works fine. I just tested on Debian Stretch and Buster yesterday, which basically is:

apt install python-virtualenv python-dev libmariadb-dev libffi-dev libssl-dev # the last two required on non-x86 only
make build

and Debian Stretch should match Ubuntu 16.04 pretty much. I didn't try make test to be true, not sure if this invokes anything in a different way.

Do build from scratch or try to migrate an existing instance? Just to rule out any left cached files etc and as aside of the syncserver.ini and in case database file, everything can be removed.

I see, thank you for the explanation. The mentioned packages are installed on my server, too. I tried both upgrading my existing installation which was working so far (due to the issues with Firefox 80 and OAuth) and with a completely fresh clone of the repo. Both cases resulted in the aforementioned error.

JFYI: I tried to reproduce the issue on a vanilla Ubuntu 20.04 installation, but there I cannot even get the make script running because virtualenv dropped the deprecated --no-site-packages option since version 20.0.0.

$ sudo apt install git make python-dev curl
$ curl https://bootstrap.pypa.io/get-pip.py --output get-pip.py
$ sudo python2 get-pip.py
$ sudo pip install virtualenv
$ git clone https://github.com/mozilla-services/syncserver.git
$ cd syncserver
$ make build

`which python2 python2.7 python | head -n 1` -m virtualenv --python=`which python2 python2.7 python | head -n 1` --no-site-packages ./local
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--app-data APP_DATA] [--reset-app-data] [--upgrade-embed-wheels] [--discovery {builtin}] [-p py] [--creator {builtin,cpython2-posix}] [--seeder {app-data,pip}] [--no-seed]
                  [--activators comma_sep_list] [--clear] [--system-site-packages] [--symlinks | --copies] [--no-download | --download] [--extra-search-dir d [d ...]] [--pip version] [--wheel version] [--setuptools version] [--no-pip]
                  [--no-wheel] [--no-setuptools] [--no-periodic-update] [--symlink-app-data] [--prompt prompt] [-h]
                  dest
virtualenv: error: unrecognized arguments: --no-site-packages
SystemExit: 2
make: *** [Makefile:27: local/COMPLETE] Error 2

On Debian Bullseye the issue is even more fundamental, since it does not ship Python 2 anymore via APT repo 😄. Did you try to use the python-virtualenv APT package instead of pulling it via pip? It is not required to install pip system wide, it is included in the virtualenv itself, and probably the APT package version is a bid older and still supports --no-site-packages.

Seems to have been removed with v20: pypa/virtualenv#1681

Ehm, now I see that Ubuntu bionic (18.04) is the last that ships Python 2 packages, can't see them anymore. How did you manage to install python-dev via APT on focal (20.04)?

In principle, Ubuntu 20.04 also does not support Python 2 anymore. However, some Python 2 packages (e.g., python-dev > python-dev-is-python2 > python2-dev) are still available via the official package repositories (universe) due to dependencies of some other packages. virtualenv is not available anymore via the official package repositories, this is why I installed it via pip (which by itself is also not available anymore via the official package repositories).

Ah nice, I didn't recognise that there is a python2 package now, same on Debian Bullseye: https://packages.debian.org/search?suite=bullseye&searchon=names&keywords=python2
No packages but it allows to pull pip via https://bootstrap.pypa.io/get-pip.py and go on from there, as you did, aside of the --no-site-packages option use. That is an own issue:

syncserver/Makefile

Line 27 in 2b8d19e

$(VIRTUALENV) --no-site-packages $(ENV)

@rfk might I ping you here to check whether this deprecated option --no-site-packages can be removed/replaced to gain virtualenv v20 compatibility?

root@VM-Stretch:/tmp/syncserver-master# virtualenv --version
1.9
root@VM-Stretch:/tmp/syncserver-master# virtualenv --help
  --no-site-packages    Don't give access to the global site-packages dir to
                        the virtual environment (default)
root@VM-Stretch:/tmp/syncserver-master# virtualenv --version
1.11
root@VM-Stretch:/tmp/syncserver-master# virtualenv --help
  --no-site-packages    DEPRECATED. Retained only for backward compatibility.
                        Not having access to global site-packages is now the
                        default behavior.

Default already since virtualenv v1.9 and deprecated since v1.11. Not sure which version you want to still support but looks like the flag can be simply skipped.
PR up to remove this flag and enable virtualenv v20+ support: #244

However all not related to your actual issue on Ubuntu 16.04. I'll at least run another test with make test on Debian Stretch and different ways of installing Python 2 + modules, let's see if we can narrow it down. I simply assume that it is pretty similar to Ubuntu 16.04, as I don't have an Ubuntu test system here 😉.

All Debian APT packages

apt update
apt install python-virtualenv python-dev g++ make libmariadb-dev
wget https://github.com/mozilla-services/syncserver/archive/master.tar.gz
tar xf master.tar.gz
rm master.tar.gz
cd syncserver-master
make test
...
OK (skipped=6)

🈯 No issue. The setuptools install is done without any further output:

2020-09-16 16:14:04 root@VM-Stretch:/tmp/syncserver-master# make test
`which python2 python2.7 python | head -n 1` -m virtualenv --python=`which python2 python2.7 python | head -n 1` --no-site-packages ./local
Already using interpreter /usr/bin/python2
New python executable in /tmp/syncserver-master/local/bin/python2
Also creating executable in /tmp/syncserver-master/local/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

Global Python version (vs 2.7.11 on Ubuntu 16.04, 2.7.12 on xenial-updates + xenial-security components):

2020-09-16 16:23:46 root@VM-Stretch:/tmp/syncserver-master# dpkg -l | grep python
ii  libpython-dev:amd64        2.7.13-2                          amd64        header files and a static library for Python (default)
ii  libpython-stdlib:amd64     2.7.13-2                          amd64        interactive high-level object-oriented language (default python version)
ii  libpython2.7:amd64         2.7.13-2+deb9u4                   amd64        Shared Python runtime library (version 2.7)
ii  libpython2.7-dev:amd64     2.7.13-2+deb9u4                   amd64        Header files and a static library for Python (v2.7)
ii  libpython2.7-minimal:amd64 2.7.13-2+deb9u4                   amd64        Minimal subset of the Python language (version 2.7)
ii  libpython2.7-stdlib:amd64  2.7.13-2+deb9u4                   amd64        Interactive high-level object-oriented language (standard library, version 2.7)
ii  python                     2.7.13-2                          amd64        interactive high-level object-oriented language (default version)
ii  python-dev                 2.7.13-2                          amd64        header files and a static library for Python (default)
ii  python-minimal             2.7.13-2                          amd64        minimal subset of the Python language (default version)
ii  python-pip-whl             9.0.1-2+deb9u2                    all          Python package installer
ii  python-pkg-resources       33.1.1-1                          all          Package Discovery and Resource Access using pkg_resources
ii  python-virtualenv          15.1.0+ds-1                       all          Python virtual environment creator
ii  python2.7                  2.7.13-2+deb9u4                   amd64        Interactive high-level object-oriented language (version 2.7)
ii  python2.7-dev              2.7.13-2+deb9u4                   amd64        Header files and a static library for Python (v2.7)
ii  python2.7-minimal          2.7.13-2+deb9u4                   amd64        Minimal subset of the Python language (version 2.7)

🈯 No issue when bootstrapping pip and installing virtualenv==16.7.10, the latest version that still supports --no-site-packages.

Aside of Python core and virtualenv, everything else should not matter as it's installed within the virtualenv only, right? So I'm a bid puzzled how a setuptools with Python >=3.5 can be pulled. Probably even only a temporary issue with thy pypa repo or wherever the virtualenv module pulls setuptools, pip and wheel from?

In case you can work with a Docker container, in order to ease debugging, I've prepared a Docker image for you with which you can reproduce the issue and experiment:

docker pull dignative/mozillasync-issue239:latest
docker run -it dignative/mozillasync-issue239:latest
git clone https://github.com/mozilla-services/syncserver.git
cd syncserver
make build

The image is based on Ubuntu 16.04 and it exhibits the same error as I could observe on my server:

# make build
`which python2 python2.7 python | head -n 1` -m virtualenv --python=`which python2 python2.7 python | head -n 1` --no-site-packages ./local
Already using interpreter /usr/bin/python2
New python executable in /syncserver/local/bin/python2
Also creating executable in /syncserver/local/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
CFLAGS="-Wno-error -Wno-error=format-security" ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future ./local/bin/pip install -i https://pypi.python.org/simple -U pip
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Looking in indexes: https://pypi.python.org/simple
Requirement already up-to-date: pip in ./local/lib/python2.7/site-packages (20.2.3)
CFLAGS="-Wno-error -Wno-error=format-security" ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future ./local/bin/pip install -r requirements.txt
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting https://github.com/mozilla-services/tokenserver/archive/bf5f232ed78fb4eb89909ec5be40f135945aa514.zip (from -r requirements.txt (line 15))
  Downloading https://github.com/mozilla-services/tokenserver/archive/bf5f232ed78fb4eb89909ec5be40f135945aa514.zip
     - 118 kB 678 kB/s
Collecting https://github.com/mozilla-services/server-syncstorage/archive/d370a488155adeb80ee6f1bc016a4aa9d009f181.zip (from -r requirements.txt (line 16))
  Downloading https://github.com/mozilla-services/server-syncstorage/archive/d370a488155adeb80ee6f1bc016a4aa9d009f181.zip
     | 155 kB 3.1 MB/s
Collecting cornice==0.16.2
  Downloading cornice-0.16.2.tar.gz (45 kB)
     |################################| 45 kB 986 kB/s 
Collecting gunicorn==19.6
  Downloading gunicorn-19.6.0-py2.py3-none-any.whl (114 kB)
     |################################| 114 kB 3.9 MB/s 
Collecting pyramid==1.5.3
  Downloading pyramid-1.5.3-py2.py3-none-any.whl (550 kB)
     |################################| 550 kB 15.5 MB/s 
Collecting WebOb==1.4.1
  Downloading WebOb-1.4.1.tar.gz (671 kB)
     |################################| 671 kB 15.9 MB/s 
Collecting requests==2.20
  Downloading requests-2.20.0-py2.py3-none-any.whl (60 kB)
     |################################| 60 kB 8.5 MB/s 
Collecting SQLAlchemy==1.3.3
  Downloading SQLAlchemy-1.3.3.tar.gz (5.9 MB)
     |################################| 5.9 MB 10.3 MB/s 
Collecting unittest2==1.1
  Downloading unittest2-1.1.0-py2.py3-none-any.whl (96 kB)
     |################################| 96 kB 7.4 MB/s 
Collecting zope.component==4.2.1
  Downloading zope.component-4.2.1.tar.gz (462 kB)
     |################################| 462 kB 13.0 MB/s 
Collecting configparser==3.5
  Downloading configparser-3.5.0.tar.gz (39 kB)
Collecting mozsvc==0.9
  Downloading mozsvc-0.9.tar.gz (37 kB)
Collecting futures==3.0
  Downloading futures-3.0.0-py2-none-any.whl (13 kB)
Collecting soupsieve==1.9.5
  Downloading soupsieve-1.9.5-py2.py3-none-any.whl (33 kB)
Collecting umemcache==1.6.3
  Downloading umemcache-1.6.3.zip (26 kB)
Collecting google-cloud-spanner==1.18.0
  Downloading google_cloud_spanner-1.18.0-py2.py3-none-any.whl (255 kB)
     |################################| 255 kB 23.0 MB/s 
Collecting alembic==1.0.9
  Downloading alembic-1.0.9.tar.gz (1.0 MB)
     |################################| 1.0 MB 18.1 MB/s 
Collecting asn1crypto==0.24.0
  Downloading asn1crypto-0.24.0-py2.py3-none-any.whl (101 kB)
     |################################| 101 kB 14.0 MB/s 
Collecting boto==2.49.0
  Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB)
     |################################| 1.4 MB 22.0 MB/s 
Collecting certifi==2019.3.9
  Downloading certifi-2019.3.9-py2.py3-none-any.whl (158 kB)
     |################################| 158 kB 38.3 MB/s 
Collecting chardet==3.0.4
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     |################################| 133 kB 40.6 MB/s 
Collecting cryptography==2.6.1
  Downloading cryptography-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl (2.3 MB)
     |################################| 2.3 MB 42.3 MB/s 
Collecting enum34==1.1.6
  Downloading enum34-1.1.6-py2-none-any.whl (12 kB)
Collecting gevent==1.4.0
  Downloading gevent-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl (5.0 MB)
     |################################| 5.0 MB 21.4 MB/s 
Collecting greenlet==0.4.13
  Downloading greenlet-0.4.13-cp27-cp27mu-manylinux1_x86_64.whl (41 kB)
     |################################| 41 kB 391 kB/s 
Collecting hawkauthlib==2.0.0
  Downloading hawkauthlib-2.0.0-py2.py3-none-any.whl (32 kB)
Collecting hupper==1.6.1
  Downloading hupper-1.6.1-py2.py3-none-any.whl (23 kB)
Collecting idna==2.8
  Downloading idna-2.8-py2.py3-none-any.whl (58 kB)
     |################################| 58 kB 8.3 MB/s 
Collecting ipaddress==1.0.22
  Downloading ipaddress-1.0.22-py2.py3-none-any.whl (18 kB)
Collecting konfig==1.1
  Downloading konfig-1.1.tar.gz (9.0 kB)
Collecting Mako==1.0.9
  Downloading Mako-1.0.9.tar.gz (459 kB)
     |################################| 459 kB 22.4 MB/s 
Collecting MarkupSafe==1.1.1
  Downloading MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl (24 kB)
Collecting mysqlclient==1.4.6
  Downloading mysqlclient-1.4.6.tar.gz (85 kB)
     |################################| 85 kB 5.2 MB/s 
Collecting Paste==3.0.8
  Downloading Paste-3.0.8-py2.py3-none-any.whl (592 kB)
     |################################| 592 kB 13.5 MB/s 
Collecting PasteDeploy==2.0.1
  Downloading PasteDeploy-2.0.1-py2.py3-none-any.whl (17 kB)
Collecting plaster==1.0
  Downloading plaster-1.0-py2.py3-none-any.whl (14 kB)
Collecting plaster-pastedeploy==0.7
  Downloading plaster_pastedeploy-0.7-py2.py3-none-any.whl (7.8 kB)
Collecting PyBrowserID==0.14.0
  Downloading PyBrowserID-0.14.0-py2-none-any.whl (49 kB)
     |################################| 49 kB 5.0 MB/s 
Collecting pycparser==2.19
  Downloading pycparser-2.19.tar.gz (158 kB)
     |################################| 158 kB 21.3 MB/s 
Collecting PyFxA==0.7.7
  Downloading PyFxA-0.7.7.tar.gz (39 kB)
Collecting PyJWT==1.7.1
  Downloading PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
Collecting PyMySQL==0.9.3
  Downloading PyMySQL-0.9.3-py2.py3-none-any.whl (47 kB)
     |################################| 47 kB 5.6 MB/s 
Collecting pymysql-sa==1.0
  Downloading pymysql_sa-1.0.tar.gz (2.7 kB)
Collecting python-dateutil==2.8.0
  Downloading python_dateutil-2.8.0-py2.py3-none-any.whl (226 kB)
     |################################| 226 kB 17.0 MB/s 
Collecting python-editor==1.0.4
  Downloading python_editor-1.0.4-py2-none-any.whl (4.9 kB)
Collecting repoze.lru==0.7
  Downloading repoze.lru-0.7.tar.gz (19 kB)
Collecting simplejson==3.16.0
  Downloading simplejson-3.16.0.tar.gz (81 kB)
     |################################| 81 kB 10.7 MB/s 
Collecting six==1.14.0
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting testfixtures==6.7.0
  Downloading testfixtures-6.7.0-py2.py3-none-any.whl (85 kB)
     |################################| 85 kB 4.1 MB/s 
Collecting tokenlib==2.0.0
  Downloading tokenlib-2.0.0-py2.py3-none-any.whl (18 kB)
Collecting translationstring==1.3
  Downloading translationstring-1.3-py2.py3-none-any.whl (15 kB)
Collecting urllib3==1.25.2
  Downloading urllib3-1.25.2-py2.py3-none-any.whl (150 kB)
     |################################| 150 kB 29.1 MB/s 
Collecting venusian==1.2.0
  Downloading venusian-1.2.0-py2.py3-none-any.whl (33 kB)
Collecting zope.deprecation==4.4.0
  Downloading zope.deprecation-4.4.0-py2.py3-none-any.whl (10 kB)
Collecting zope.interface==4.6.0
  Downloading zope.interface-4.6.0-cp27-cp27mu-manylinux1_x86_64.whl (164 kB)
     |################################| 164 kB 17.1 MB/s 
Collecting cffi==1.14.0
  Downloading cffi-1.14.0-cp27-cp27mu-manylinux1_x86_64.whl (387 kB)
     |################################| 387 kB 19.5 MB/s 
Collecting pyramid_hawkauth
  Downloading pyramid_hawkauth-2.0.0-py2.py3-none-any.whl (21 kB)
Collecting wsgiproxy
  Downloading WSGIProxy-0.2.2.tar.gz (10 kB)
Collecting webtest
  Downloading WebTest-2.0.35-py2.py3-none-any.whl (32 kB)
Requirement already satisfied: setuptools in ./local/lib/python2.7/site-packages (from pyramid==1.5.3->-r requirements.txt (line 3)) (45.0.0)
ERROR: Package 'setuptools' requires a different Python: 2.7.12 not in '>=3.5'
Makefile:27: recipe for target 'local/COMPLETE' failed
make: *** [local/COMPLETE] Error 1

So the issue seems not to be related to the specific configuration of my server. For your reference, this is the Dockerfile I wrote for the image:

FROM ubuntu:16.04
ENV DEBIAN_FRONTEND noninteractive
ENV HOME /root

RUN apt-get update && apt-get -y install apt-utils
RUN apt-get update && apt-get -y install \
    curl \
    git \
    libmariadb-client-lgpl-dev \
    libmysqlclient-dev \
    make \
    python-dev \
    python-virtualenv

Good to have a clean testing environment. Good to know the error actually happened at a different stage than I thought:

Installing setuptools, pkg_resources, pip, wheel...done.

works fine at first, but then later fails after verifying it's there as pyramid dependency.

I just compared the exact build output with mine on Debian Stretch and the installed setuptools version is strangely a different one:

Requirement already satisfied: setuptools in ./local/lib/python2.7/site-packages (from pyramid==1.5.3->-r requirements.txt (line 3)) (44.1.1)

44.1.1 vs 45.0.0 in all your cases. And now I remember seeing something similar in another context, let me check.

EDIT: Here you go: MichaIng/DietPi#3346
Basically same issue: setuptools 45.0.0 is pulled, which is, other than the name suggests (setuptools-45.0.0-py2.py3-none-any.whl) Python 3 only. Two issues came together:

A pre-compiled wheel must not have py2.py3 naming when it's Python 3 only.
https://www.piwheels.org/ did not read the Requires-Python attribute which does contain the info that it is Python 3 only as well.
piwheels is used on RPi, and they implemented reading the flag now: piwheels/piwheels#208

So now we're back on the initial setuptools install made by virtualenv with hidden output. Does it have some verbose flag to see where it pulls setuptools from (according to low install time, it seems to pull pre-compiled wheels)? Yes it has.
@DigNative try the following:

mkdir /tmp/venv_test
python2 -m virtualenv -v --python=python2 --no-site-packages /tmp/venv_test

To compare, my result:

2020-09-17 15:32:17 root@VM-Stretch:~# python2 -m virtualenv -v --python=python2 --no-site-packages /tmp/venv_test
Already using interpreter /usr/bin/python2
Creating /tmp/venv_test/lib/python2.7
Symlinking Python bootstrap modules
  Symlinking /tmp/venv_test/lib/python2.7/lib-dynload
  Symlinking /tmp/venv_test/lib/python2.7/os.py
  Ignoring built-in bootstrap module: posix
  Symlinking /tmp/venv_test/lib/python2.7/posixpath.py
  Cannot import bootstrap module: nt
  Symlinking /tmp/venv_test/lib/python2.7/ntpath.py
  Symlinking /tmp/venv_test/lib/python2.7/genericpath.py
  Symlinking /tmp/venv_test/lib/python2.7/fnmatch.py
  Symlinking /tmp/venv_test/lib/python2.7/locale.py
  Symlinking /tmp/venv_test/lib/python2.7/encodings
  Symlinking /tmp/venv_test/lib/python2.7/codecs.py
  Symlinking /tmp/venv_test/lib/python2.7/stat.py
  Symlinking /tmp/venv_test/lib/python2.7/UserDict.py
  Symlinking /tmp/venv_test/lib/python2.7/copy_reg.py
  Symlinking /tmp/venv_test/lib/python2.7/types.py
  Symlinking /tmp/venv_test/lib/python2.7/re.py
  Symlinking /tmp/venv_test/lib/python2.7/sre.py
  Symlinking /tmp/venv_test/lib/python2.7/sre_parse.py
  Symlinking /tmp/venv_test/lib/python2.7/sre_constants.py
  Symlinking /tmp/venv_test/lib/python2.7/sre_compile.py
  Ignoring built-in bootstrap module: zlib
  Symlinking /tmp/venv_test/lib/python2.7/warnings.py
  Symlinking /tmp/venv_test/lib/python2.7/linecache.py
  Symlinking /tmp/venv_test/lib/python2.7/_abcoll.py
  Symlinking /tmp/venv_test/lib/python2.7/abc.py
  Symlinking /tmp/venv_test/lib/python2.7/_weakrefset.py
Creating /tmp/venv_test/lib/python2.7/site-packages
Writing /tmp/venv_test/lib/python2.7/site.py
Writing /tmp/venv_test/lib/python2.7/orig-prefix.txt
Writing /tmp/venv_test/lib/python2.7/no-global-site-packages.txt
Creating parent directories for /tmp/venv_test/include
Symlinking /tmp/venv_test/include/python2.7
Creating /tmp/venv_test/bin
New python executable in /tmp/venv_test/bin/python2
Changed mode of /tmp/venv_test/bin/python2 to 0755
Also creating executable in /tmp/venv_test/bin/python
Changed mode of /tmp/venv_test/bin/python to 0755
Testing executable with /tmp/venv_test/bin/python2 -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
Got sys.prefix result: u'/tmp/venv_test'
Symlinking /tmp/venv_test/local/bin
Symlinking /tmp/venv_test/local/include
Symlinking /tmp/venv_test/local/lib
Creating /tmp/venv_test/lib/python2.7/distutils
Writing /tmp/venv_test/lib/python2.7/distutils/__init__.py
Writing /tmp/venv_test/lib/python2.7/distutils/distutils.cfg
Installing setuptools, pkg_resources, pip, wheel...
  Collecting setuptools
    Using cached https://files.pythonhosted.org/packages/e1/b7/182161210a13158cd3ccc41ee19aadef54496b74f2817cc147006ec932b4/setuptools-44.1.1-py2.py3-none-any.whl
  Collecting pkg_resources
  Collecting pip
    Cache entry deserialization failed, entry ignored
    Cache entry deserialization failed, entry ignored
    Using cached https://files.pythonhosted.org/packages/4e/5f/528232275f6509b1fff703c9280e58951a81abe24640905de621c9f81839/pip-20.2.3-py2.py3-none-any.whl
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/a7/00/3df031b3ecd5444d572141321537080b40c1c25e1caa3d86cdd12e5e919c/wheel-0.35.1-py2.py3-none-any.whl
  Installing collected packages: setuptools, pkg-resources, pip, wheel
  Successfully installed pip-20.2.3 pkg-resources-0.0.0 setuptools-44.1.1 wheel-0.35.1
...Installing setuptools, pkg_resources, pip, wheel...done.
Writing /tmp/venv_test/bin/activate
Writing /tmp/venv_test/bin/activate.fish
Writing /tmp/venv_test/bin/activate_this.py
Writing /tmp/venv_test/bin/activate.csh
Writing /tmp/venv_test/bin/python-config
Changed mode of /tmp/venv_test/bin/python-config to 0755

Here it collects setuptools with correct version 44.1.1:

https://files.pythonhosted.org/packages/e1/b7/182161210a13158cd3ccc41ee19aadef54496b74f2817cc147006ec932b4/setuptools-44.1.1-py2.py3-none-any.whl

@rfk might I ping you here to check whether this deprecated option --no-site-packages can be removed/replaced
to gain virtualenv v20 compatibility?

Sorry I didn't get a chance to reply to this earlier, but 👍 to removing this; I recall having to make as similar change in one of the upstream repos recently as well.

All fine, was easy enough to commit directly, thanks for merging so quickly.

As of the actual issue here, did you ever faced such, a Python 3 only module/version pulled by Python 2 virtualenv or pip? If the case matches, probably the example of how such issues are now prevented in the first place on piwheels.org can be suggested to other Python repositories.

@MichaIng Thank you for your extensive testing and debugging! I tried what you suggested on my server:

$ mkdir /tmp/venv_test
$ python2 -m virtualenv -v --python=python2 --no-site-packages /tmp/venv_test
Already using interpreter /usr/bin/python2
Creating /tmp/venv_test/lib/python2.7
Symlinking Python bootstrap modules
  Symlinking /tmp/venv_test/lib/python2.7/lib-dynload
  Symlinking /tmp/venv_test/lib/python2.7/os.py
  Ignoring built-in bootstrap module: posix
  Symlinking /tmp/venv_test/lib/python2.7/posixpath.py
  Cannot import bootstrap module: nt
  Symlinking /tmp/venv_test/lib/python2.7/ntpath.py
  Symlinking /tmp/venv_test/lib/python2.7/genericpath.py
  Symlinking /tmp/venv_test/lib/python2.7/fnmatch.py
  Symlinking /tmp/venv_test/lib/python2.7/locale.py
  Symlinking /tmp/venv_test/lib/python2.7/encodings
  Symlinking /tmp/venv_test/lib/python2.7/codecs.py
  Symlinking /tmp/venv_test/lib/python2.7/stat.py
  Symlinking /tmp/venv_test/lib/python2.7/UserDict.py
  Symlinking /tmp/venv_test/lib/python2.7/copy_reg.py
  Symlinking /tmp/venv_test/lib/python2.7/types.py
  Symlinking /tmp/venv_test/lib/python2.7/re.py
  Symlinking /tmp/venv_test/lib/python2.7/sre.py
  Symlinking /tmp/venv_test/lib/python2.7/sre_parse.py
  Symlinking /tmp/venv_test/lib/python2.7/sre_constants.py
  Symlinking /tmp/venv_test/lib/python2.7/sre_compile.py
  Ignoring built-in bootstrap module: zlib
  Symlinking /tmp/venv_test/lib/python2.7/warnings.py
  Symlinking /tmp/venv_test/lib/python2.7/linecache.py
  Symlinking /tmp/venv_test/lib/python2.7/_abcoll.py
  Symlinking /tmp/venv_test/lib/python2.7/abc.py
  Symlinking /tmp/venv_test/lib/python2.7/_weakrefset.py
Creating /tmp/venv_test/lib/python2.7/site-packages
Writing /tmp/venv_test/lib/python2.7/site.py
Writing /tmp/venv_test/lib/python2.7/orig-prefix.txt
Writing /tmp/venv_test/lib/python2.7/no-global-site-packages.txt
Creating parent directories for /tmp/venv_test/include
Symlinking /tmp/venv_test/include/python2.7
Creating /tmp/venv_test/bin
New python executable in /tmp/venv_test/bin/python2
Changed mode of /tmp/venv_test/bin/python2 to 0775
Also creating executable in /tmp/venv_test/bin/python
Changed mode of /tmp/venv_test/bin/python to 0775
Testing executable with /tmp/venv_test/bin/python2 -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
Got sys.prefix result: u'/tmp/venv_test'
Symlinking /tmp/venv_test/local/bin
Symlinking /tmp/venv_test/local/lib
Symlinking /tmp/venv_test/local/include
Creating /tmp/venv_test/lib/python2.7/distutils
Writing /tmp/venv_test/lib/python2.7/distutils/__init__.py
Writing /tmp/venv_test/lib/python2.7/distutils/distutils.cfg
Installing setuptools, pip, wheel...
  Collecting setuptools
    Downloading https://files.pythonhosted.org/packages/af/e7/02db816dc88c598281bacebbb7ccf2c9f1a6164942e88f1a0fded8643659/setuptools-45.0.0-py2.py3-none-any.whl (583kB)
  Collecting pip
    Downloading https://files.pythonhosted.org/packages/4e/5f/528232275f6509b1fff703c9280e58951a81abe24640905de621c9f81839/pip-20.2.3-py2.py3-none-any.whl (1.5MB)
  Collecting wheel
    Downloading https://files.pythonhosted.org/packages/a7/00/3df031b3ecd5444d572141321537080b40c1c25e1caa3d86cdd12e5e919c/wheel-0.35.1-py2.py3-none-any.whl
  Installing collected packages: setuptools, pip, wheel
  Successfully installed pip-20.2.3 setuptools-45.0.0 wheel-0.35.1
...Installing setuptools, pip, wheel...done.
Writing /tmp/venv_test/bin/activate
Writing /tmp/venv_test/bin/activate.fish
Writing /tmp/venv_test/bin/activate_this.py
Writing /tmp/venv_test/bin/activate.csh
Writing /tmp/venv_test/bin/python-config
Changed mode of /tmp/venv_test/bin/python-config to 0775

So, in my case, setuptools-45.0.0 is still pulled for whatever reason. I tried the same with the Docker image that I provided and I can observe the same behavior there.

The command is only what make build does under the hood and more verbose. It was to find out the repository where the faulty setuptools version is pulled from 😉: https://files.pythonhosted.org

That is the bug same mirror: pypa/virtualenv#1493
~~Or here, which shows that it is the exact same bug was with piwheels.org before they implemented reading the Requires-Python flag: pypa/pip#7586~~

The problem now is that I cannot see any contact or issue tracker for pythonhosted.org where we need to report this. PyPI is not responsible (AFAIK). Although whois says "Organization: Python Software Foundation"

Ah lol actually in make case the mirror was the same. So there seems to be a difference on another layer between our systems then. I'm a bid puzzled. Let's see if someone in the linked issue has an idea: pypa/virtualenv#1493 (comment)

Could you try it this works? 0bc618a
This can be probably done more beautiful, or combined with the explicit pip install/upgrade from a specific index, that is already in place, probably a workaround for a similar issue. And I didn't test it 😉.

Explicit pip install/upgrade: d03e4ec#diff-b67911656ef5d18c4ae36cb6741b7965
Not sire why this was required as virtualenv installs the latest version as well? Or is it to upgrade in case the virtual environment already exists?

Thank you! I can confirm that using 0bc618a works for me (N.B.: This commit is not referenced in the Git repository by any branch/tag, this is why I had to manually fetch it.). I am just receiving an "error" (I think it's just a notice despite of the displayed word "error" since both make build and make test complete successfully):

$ make build
[...]
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

gevent 1.4.0 requires greenlet>=0.4.14; platform_python_implementation == "CPython", but you'll have greenlet 0.4.13 which is incompatible.
google-api-core 1.22.2 requires futures>=3.2.0; python_version < "3.2", but you'll have futures 3.0.0 which is incompatible.
tokenserver 1.5.11 requires configparser==3.7.4, but you'll have configparser 3.5.0 which is incompatible.
tokenserver 1.5.11 requires cornice==3.5.1, but you'll have cornice 0.16.2 which is incompatible.
tokenserver 1.5.11 requires gunicorn==19.10.0, but you'll have gunicorn 19.6.0 which is incompatible.
tokenserver 1.5.11 requires mozsvc==0.10, but you'll have mozsvc 0.9 which is incompatible.
tokenserver 1.5.11 requires pyramid==1.10.4, but you'll have pyramid 1.5.3 which is incompatible.
tokenserver 1.5.11 requires requests==2.22.0, but you'll have requests 2.20.0 which is incompatible.
tokenserver 1.5.11 requires WebOb==1.8.5, but you'll have webob 1.4.1 which is incompatible.
[...]
Adding syncserver 1.9.1 to easy-install.pth file

Installed /t/syncserver-0bc618a76cffe3c0af9144e852693ed7af109ed1
Processing dependencies for syncserver==1.9.1
Finished processing dependencies for syncserver==1.9.1
touch ./local/COMPLETE

$ make test
CFLAGS="-Wno-error -Wno-error=format-security" ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future ./local/bin/pip install -r dev-requirements.txt
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting flake8==3.3
  Downloading flake8-3.3.0-py2.py3-none-any.whl (66 kB)
     |################################| 66 kB 1.2 MB/s 
Collecting nose==1.3.7
  Downloading nose-1.3.7-py2-none-any.whl (154 kB)
     |################################| 154 kB 5.2 MB/s 
Collecting pycodestyle<2.4.0,>=2.0.0
  Downloading pycodestyle-2.3.1-py2.py3-none-any.whl (45 kB)
     |################################| 45 kB 4.6 MB/s 
Requirement already satisfied: configparser; python_version < "3.2" in ./local/lib/python2.7/site-packages (from flake8==3.3->-r dev-requirements.txt (line 1)) (3.5.0)
Collecting pyflakes<1.6.0,>=1.5.0
  Downloading pyflakes-1.5.0-py2.py3-none-any.whl (225 kB)
     |################################| 225 kB 9.5 MB/s 
Collecting mccabe<0.7.0,>=0.6.0
  Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Requirement already satisfied: enum34; python_version < "3.4" in ./local/lib/python2.7/site-packages (from flake8==3.3->-r dev-requirements.txt (line 1)) (1.1.6)
Installing collected packages: pycodestyle, pyflakes, mccabe, flake8, nose
Successfully installed flake8-3.3.0 mccabe-0.6.1 nose-1.3.7 pycodestyle-2.3.1 pyflakes-1.5.0
./local/bin/flake8 ./syncserver
./local/bin/nosetests -s syncstorage.tests
/t/syncserver-0bc618a76cffe3c0af9144e852693ed7af109ed1/local/local/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.2) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
.............................................................................S...S...SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.....................................................S......................................................................SSSSSSSSSSSSSSSSS.SS/t/syncserver-0bc618a76cffe3c0af9144e852693ed7af109ed1/local/local/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.2) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
/t/syncserver-0bc618a76cffe3c0af9144e852693ed7af109ed1/local/local/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.2) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
.........................
----------------------------------------------------------------------
Ran 446 tests in 200.292s

OK (SKIP=214)
# Tokenserver tests currently broken due to incorrect file paths
# ./local/bin/nosetests -s tokenserver.tests
# Test against a running server.
./local/bin/gunicorn --paste syncserver/tests.ini 2> /dev/null & SERVER_PID=$!; \
sleep 2; \
./local/bin/python -m syncstorage.tests.functional.test_storage \
	--use-token-server http://localhost:5000/token/1.0/sync/1.5; \
kill $SERVER_PID
/t/syncserver-0bc618a76cffe3c0af9144e852693ed7af109ed1/local/local/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.2) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
test_accessing_info_collections_with_an_expired_token (syncstorage.tests.functional.support.LiveTestCases) ... skipped ''
test_alternative_formats (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_app_newlines_when_payloads_contain_newlines (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_bad_cache (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_batch_empty_commit (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_batch_id_is_correctly_scoped_to_a_collection (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_batch_id_is_correctly_scoped_to_a_user (syncstorage.tests.functional.support.LiveTestCases) ... skipped 'Skipped when testing a live server'
test_batch_partial_update (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_batch_size_limits (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_batch_ttl_is_based_on_commit_timestamp (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_batch_ttl_update (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_batch_uploads_properly_update_info_collections (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_batch_with_failing_bsos (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_batch_with_immediate_commit (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_batches (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_bulk_update_of_ttls_without_sending_data (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_collection_usage (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_create_bso_with_null_ttl (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_delete_collection_items (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_delete_item (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_delete_storage (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_get_collection (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_get_collection_count (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_get_collection_ttl (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_get_info_collections (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_get_item (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_guid_deletion (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_handling_of_invalid_bso_fields (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_handling_of_invalid_json_in_bso_uploads (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_if_modified_since_on_info_views (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_ifunmodifiedsince (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_meta_global_sanity (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_multi_item_post_limits (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_overquota (syncstorage.tests.functional.support.LiveTestCases) ... skipped ''
test_pagination_with_newer_and_sort_by_oldest (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_pagination_with_older_and_sort_by_newest (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_quota (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_rejection_of_known_bad_payloads (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_set_collection (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_set_collection_input_formats (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_set_collection_with_if_modified_since (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_set_item (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_set_item_input_formats (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_specifying_ids_with_percent_encoded_query_string (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_strict_newer (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_strict_older (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_that_404_responses_have_a_json_body (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_that_batch_deletes_are_limited_to_max_number_of_ids (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_that_batch_gets_are_limited_to_max_number_of_ids (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_that_bsos_can_have_a_collection_field (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_that_expired_items_can_be_overwritten_via_PUT (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_that_internal_server_fields_are_not_echoed (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_that_negative_integer_fields_are_not_accepted (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_that_we_dont_resurrect_committed_batches (syncstorage.tests.functional.support.LiveTestCases) ... skipped 'failed to trigger re-use of batchid'
test_that_x_last_modified_is_sent_for_all_get_requests (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_timestamp_numbers_are_decimals (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_update_of_ttl_without_sending_data (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_users_with_the_same_batch_id_get_separate_data (syncstorage.tests.functional.support.LiveTestCases) ... skipped 'Skipped when testing a live server'
test_we_dont_need_no_stinkin_batches (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_weird_args (syncstorage.tests.functional.support.LiveTestCases) ... ok
test_x_timestamp_header (syncstorage.tests.functional.support.LiveTestCases) ... skipped ''

----------------------------------------------------------------------
Ran 61 tests in 30.104s

OK (skipped=6)

Great. Yes the error due to the changing way how pip resolves dependencies is another topic to take care of.

Yes I only made the commit to my own fork so far. Will open a PR: #245

I've noticed that the PR has been merged two days ago. I tried again now with the latest version of master and can confirm that the issue is resolved now for me. From my perspective, the issue can be closed.

Thank you very much for your work and the timely fix!