Unwanted files in precompiled pip package

[puntopaz]

This is related to: #659
mitogen version: v0.3.4

I stumbled on this problem while trying a python3 -m compileall -j 0 /usr/lib/python3.10.
Basically, when you install mitogen 0.3.4 using pip3, the precompiled package still contains the files listed in #659 even if they were removed in the repo.

Files found in package but not in source repo:

/usr/lib/python3.10/site-packages/mitogen/compat
└── shutil.py

/usr/lib/python3.10/site-packages/ansible_mitogen/compat
└── simplejson
    ├── __init__.py
    ├── decoder.py
    ├── encoder.py
    └── scanner.py

The problem appears both in x64 and Arm pip packages.

You can reproduce the problem building and running this Docker image, which compares package-installed and repo contents:

docker build -t mitogen-pkg-test -f Dockerfile-mitogen
docker run -it --rm mitogen-pkg-test

Dockerfile-mitogen:

FROM alpine:3.16 as builder
WORKDIR /data

# Base tools
RUN apk add \
		py3-pip \
		python3 \
		tree \
		curl \
		diffutils

# Pip required tools
RUN pip3 install \
		wheel

# Install package and remove precompiled files (for easier folder comparision)
RUN pip3 install \
		mitogen \
	&& find /usr/lib/python3.10/site-packages/ -name '__pycache__' -print0 | xargs -0 -n1 rm -rf


# Get sources from Github
RUN curl -s -L https://github.com/mitogen-hq/mitogen/archive/refs/tags/v0.3.4.tar.gz -o /tmp/mitogen-v0.3.4.tar.gz \
    && tar -xf /tmp/mitogen-v0.3.4.tar.gz -C /data \
	&& rm -f /tmp/mitogen-v0.3.4.tar.gz

# Prepare diff file
RUN touch /data/differences.txt

# Print installed version
RUN echo "pip list | grep mitogen" >> /data/differences.txt \
    && pip list | grep mitogen >> /data/differences.txt

# Detect folder differences
RUN echo >> /data/differences.txt \
    && echo "diff -r /usr/lib/python3.10/site-packages/mitogen /data/mitogen-0.3.4/mitogen" >> /data/differences.txt \
    && diff -r /usr/lib/python3.10/site-packages/mitogen         /data/mitogen-0.3.4/mitogen         >> /data/differences.txt || true
RUN echo >> /data/differences.txt \
    && echo "diff -r /usr/lib/python3.10/site-packages/ansible_mitogen /data/mitogen-0.3.4/ansible_mitogen" >> /data/differences.txt \
    && diff -r /usr/lib/python3.10/site-packages/ansible_mitogen /data/mitogen-0.3.4/ansible_mitogen >> /data/differences.txt || true

# Add tree view for mitogen subfolder
RUN echo >> /data/differences.txt \
    && echo "Tree view for mitogen/compat subfolder" >> /data/differences.txt \
    && tree /usr/lib/python3.10/site-packages/mitogen/compat >> /data/differences.txt \
    && echo >> /data/differences.txt \
    && tree /data/mitogen-0.3.4/mitogen/compat >> /data/differences.txt

# Add tree view for ansible_mitogen subfolder
RUN echo >> /data/differences.txt \
    && echo "Tree view for ansible_mitogen/compat subfolder" >> /data/differences.txt \
    && tree /usr/lib/python3.10/site-packages/ansible_mitogen/compat >> /data/differences.txt \
    && echo >> /data/differences.txt \
    && tree /data/mitogen-0.3.4/ansible_mitogen/compat >> /data/differences.txt

CMD ["/bin/cat", "/data/differences.txt"]
#CMD ["/bin/sh"]

Sample output:

ansible@ubuntu22:~/custom-docker-image$ docker run -it --rm mitogen-pkg-test
pip list | grep mitogen
mitogen        0.3.4

diff -r /usr/lib/python3.10/site-packages/mitogen /data/mitogen-0.3.4/mitogen
Only in /usr/lib/python3.10/site-packages/mitogen/compat: shutil.py

diff -r /usr/lib/python3.10/site-packages/ansible_mitogen /data/mitogen-0.3.4/ansible_mitogen
Only in /usr/lib/python3.10/site-packages/ansible_mitogen/compat: simplejson

Tree view for mitogen/compat subfolder
/usr/lib/python3.10/site-packages/mitogen/compat
├── __init__.py
├── pkgutil.py
├── shutil.py
└── tokenize.py

0 directories, 4 files

/data/mitogen-0.3.4/mitogen/compat
├── __init__.py
├── pkgutil.py
└── tokenize.py

0 directories, 3 files

Tree view for ansible_mitogen/compat subfolder
/usr/lib/python3.10/site-packages/ansible_mitogen/compat
├── __init__.py
└── simplejson
    ├── __init__.py
    ├── decoder.py
    ├── encoder.py
    └── scanner.py

1 directory, 5 files

/data/mitogen-0.3.4/ansible_mitogen/compat
└── __init__.py

0 directories, 1 file

[moreati]

0.3.5 release doesn't have those files

➜  mitogen2 git:(docs-master) zipinfo dist/mitogen-0.3.5-py2.py3-none-any.whl | grep -E 'shutil|simplejson' | wc -l
       0
➜  mitogen2 git:(docs-master) tar tf dist/mitogen-0.3.5.tar.gz  | grep -E 'shutil|simplejson' | wc -l              
       0

but the release process is still fairly manual, so it. ould happen again. Leaving open