Pipfile.lock hashes change between operating systems
Closed this issue · 20 comments
Currently if I lock a Pipfile in OSX and then try to install the lock file in linux it fails as many of the packages seem to have different versions based on the OS, with different hashes
An error occured while installing!
THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
gevent==1.2.1 from https://pypi.python.org/packages/a1/41/65eee925d5fe1e1a83ab75e2e83952515f6789aaf561a8183e4c515b49a6/gevent-1.2.1-cp27-cp27mu-manylinux1_x86_64.whl#md5=8b09f79ffb73bfa47806f2719d0b34f1 (from -r /tmp/tmpvJReOn-requirements.txt (line 6)):
Expected sha256 f68fa3b23e518ce12f8b14264ba567eb2f8a22923f85257d5043fb81e78dbc43
Got 5b9ba1abd75ed7e3d667dd52baba60e902ab0de583aa92d5f80c6fecdae06a28
greenlet==0.4.12 from https://pypi.python.org/packages/fd/b8/dd04dd0ce13ba7412eac94f6e0456e26e45d0e1df5b8019fb15832d850b0/greenlet-0.4.12-cp27-cp27mu-manylinux1_x86_64.whl#md5=1719cc1f62dd52fd5af1ee4dc38c83d2 (from -r /tmp/tmpvJReOn-requirements.txt (line 21)):
Expected sha256 e4c99c6010a5d153d481fdaf63b8a0782825c0721506d880403a3b9b82ae347e
Got 21232907c8c26838b16915bd8fbbf82fc70c996073464cc70981dd4a96bc841c
cryptography==1.7.2 from https://pypi.python.org/packages/99/df/71c7260003f5c469cec3db4c547115df39e9ce6c719a99e067ba0e78fd8a/cryptography-1.7.2.tar.gz#md5=fade66de437392ed1ba6980768626204 (from -r /tmp/tmpvJReOn-requirements.txt (line 24)):
Expected sha256 64c875d0384c8c1dc6134fc7ddcfe39b09ea9d5bb2c7616792006e83d85db52c
Got 878cb68b3da3d493ffd68f36db11c29deee623671d3287c3f8d685117ffda9a9
billiard==3.3.0.23 from https://pypi.python.org/packages/64/a6/d7b6fb7bd0a4680a41f1d4b27061c7b768c673070ba8ac116f865de4e7ca/billiard-3.3.0.23.tar.gz#md5=6ee416e1e7c8d8164ce29d7377cca6a4 (from -r /tmp/tmpvJReOn-requirements.txt (line 26)):
Expected sha256 c0cbe8d45ba8d8213ad68ef9a1881002a151569c9424d551634195a18c3a4160
Got 692a2a5a55ee39a42bcb7557930e2541da85df9ea81c6e24827f63b80cd39d0b
cffi==1.9.1 from https://pypi.python.org/packages/5f/bc/9fd2ce704560d2be1b13fe8939bca4941c8cf60e7fc97a0abd73edd44686/cffi-1.9.1-cp27-cp27mu-manylinux1_x86_64.whl#md5=dd6e35f133ad90e18cfdb44e0aa7d62b (from -r /tmp/tmpvJReOn-requirements.txt (line 48)):
Expected sha256 d3e3063af1fa6b59e255da9a812891cdaf24b90fbaf653c02797871069b7c4c9
Got 9163f7743cf9991edaddf9cf886708e288fab38e1b9fec9c41c15c85c8f7f147
Hey @frankh, we're currently only able to support lock files on similar platforms because of this. This also applies to lock files between Python 2 and 3 due to version specific wheels.
For now the suggested practice is to run pipenv lock on new environments before trying to use pipenv.
Is this something you're planning on fixing? This essentially means anyone who doesn't develop on Linux won't be able to deploy their applications without creating a new lock file for the production environment, which kind of defeats the purpose of the lock file.
Hey @mixxorz, this was originally discussed in #147. pip will automatically try to download pre-built wheels when possible which can be platform (and major version) specific. This means we'll usually be hashing files specific to that platform.
The "fix" is either to drop wheel support (which I think is very unlikely), or to download every file for a given version and keep multiple hashes. The issue with the latter is downloading every possible copy of a package from possibly multiple sources becomes unacceptably costly as project sizes increase. Trying to predict every platform a project may be moved to is something that's hard to get right with the amount of customization allowed.
As I stated in #147, I agree re-locking does somewhat defeat the purpose of the hash to have to run a lock before deploy to a specific type of architecture. Until a better solution presents itself though, I'm not sure we can do anything helpful for all users here. If you have suggestions/thoughts on specifics, I'd be happy to hear them.
Also, as I failed to document after the discussion in #147, you can create a pip.conf file on all of your deployments with no-binary=:all:. This will force pip to only download source archives which should be universal. This is not something we're going to explicitly support though, so it's a voluntary workaround for the time being.
@nateprewitt Is there any way to disable hashing? FWIW, Bundler doesn't have this problem.
@jacebrowning, that's probably a question for Kenneth. I don't think we want to disable hashes though.
As for Bundler, I'm honestly not incredibly well versed in how gems are distributed but I do know Bundler uses more specificity including a Platforms section. I haven't spent much time in Ruby which is why I'm likely missing some obvious insights here. That said, pipenv isn't intended to be a python Bundler clone either. I'll try to do some reading this weekend and see if there's anything simple we're missing.
Gemfile.lock does not include hashes of Gems installed from package repositories, but does include SHAs from dependencies installed from source. If I am unable to develop on one system and deploy on another, than Pipfile.lock serves no purpose for me.
@jacebrowning yeah, I completely get where you're coming from with that. I've been doing a bit of reading since my last post. I'll try to get some time to talk to @kennethreitz about options here, unless he has further thoughts he'd like to post.
@nateprewitt to clarify https://github.com/kennethreitz/pipenv/issues/210#issuecomment-283182256, if I'm running into hash conflicts, I should not commit Pipfile.lock (edit: or, it doesn't matter), and add run pipfile lock in my deployment script before pipenv install?
@wasabigeek this issue is quite old. You should be able to move your lockfiles between machines now. If you're experiencing issues with pipenv 8.2.7+ please open a new issue. Thanks!
@nateprewitt thanks - I got an error about the hashes, but on further inspection seems it's due to the pre-included packages in my host's images, not related to Pipenv. I'll update if I find out otherwise otherwise. Thanks!
@nateprewitt I did pip --upgrade to 8.3.1 but still encountered the following error after running my deploy script:
THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
oauthlib==1.0.3 from file:///usr/share/pip-wheels/oauthlib-1.0.3-py3-none-any.whl (from -r /tmp/pipenv-az9xn_nu-requirement.txt (line 1)):
Expected sha256 ef4bfe4663ca3b97a995860c0173b967ebd98033d02f38c9e1b2cbb6c191d9ad
Got 5454640e006c0eecce902638cd7a3fda384d083abf3f7412173fb075610e2256
I confirmed the hashes using pip download and pip hash on local and on the server, they're as above. Are there other factors that could contribute to this?
(In the end I did pipenv install <package> on the server itself)
The "fix" is either to drop wheel support (which I think is very unlikely), or to download every file for a given version and keep multiple hashes. The issue with the latter is downloading every possible copy of a package from possibly multiple sources becomes unacceptably costly as project sizes increase. Trying to predict every platform a project may be moved to is something that's hard to get right with the amount of customization allowed.
As I stated in #147, I agree re-locking does somewhat defeat the purpose of the hash to have to run a lock before deploy to a specific type of architecture. Until a better solution presents itself though, I'm not sure we can do anything helpful for all users here. If you have suggestions/thoughts on specifics, I'd be happy to hear them.
One thought. What about updating the Pipefile.lock format such that any platform-specific wheels get their own section/hashes. Meaning when another user locks on a new platform, the lock file gets updates with the new wheels and hashes them for the new platform, but also updates the platform-specific wheels already in the .lock file?
In other words, narrow down the wheels that need to be hashed to only the platforms users have locked from previously?
Similar to the previous comment ... would it work to have the ability to add the list of platforms that the user wants to lock file to work on. As stated, trying to guess would be hard but in my case, I need the lock file on two platforms.
Have the same need to generate an universal Pipfile.lock
I make development on win32 and deploy in a docker image based on a linux distribution.
Without this ability the Pipfile.lock is unusable no ?
Does anybody know a work around? This is really a bummer. Anything I've seen so far from locking in new environment, ignoring hashes or lock file completely jeopardizes the use of pipenv. What's the purpose of locking an environment, when I ignore the lock later on.
Does anybody know a work around? This is really a bummer. Anything I've seen so far from locking in new environment, ignoring hashes or lock file completely jeopardizes the use of pipenv. What's the purpose of locking an environment, when I ignore the lock later on.
I bumped into this problem 2 1/2 years ago. It was another thorn in the side of Python — one of many. As I recall, I did figure out a very complex dance to get around the problem but I can’t recall at this point what it was.
I know this isn’t what the thread or you want to hear but I would suggest abandoning Python sooner rather than later. I have dozens of very technical non-emotional reasons for saying that. They key developers, I guess, are just not very good or disciplined or something. I’m not sure but the Python ecosystem is very lacking.
@kato-m I can think of two workarounds without building from source.
- Run
pipenv lockin a container or VM to ensure it resolves your target platform's hashes. This will only work if you're trying to build for a specific target and don't care as much about your dev environment-- your dev environment will have to build from source. Not ideal, but probably ok for a lot of people. You could also use the container as your dev environment. - Generate the file as normal and add hashes for missing platforms. I think this is the ideal, but it would require a lot of automation work. There are a few ways to do this.
- The hashes are all listed on PyPi, e.g., https://pypi.org/project/cryptography/#files, so you could grab them manually or write a script to scrape them and add them.
- Like above, run
pipenv lockin different containers/VMs to generate for your target platforms, then do a hash merge to create a unified lockfile.
I hope this is reexamined, but I have nothing to contribute but ideas. I think it would be nice if pipenv lock had a non-default option to resolve hashes for alternative platforms. Like, pipenv lock --alt-platforms=manylinux_2_24_x86_64,musllinux_1_1_aarch64 or something, then it generates for the native platform plus the alt-plaforms. pipenv lock without the option could continue to work as it does now.
This is really troublesome if your dev environment is Windows and your production environment is linux.
How about adding an "platforms = " specifier to the Pipfile?
@xyxz-web You are commenting on a closed issue from near the inception of the project, in 2017. It is 2022 now, and it is troublesome to get such a low-detail comment on a long closed issue -- if you feel there is a bug or need for improvement then please open a new more detailed issue report. It is the only real way to may traction on this one. Thank you!