python-3.5.4-macosx10.9.pkg does not exist in https://www.python.org/ftp/python/3.5.4/
Closed this issue · 17 comments
Hi,
Our project's build started to fail:
++ local inst_ext=pkg
++ echo python-3.5.4-macosx10.9.pkg
-
local py_inst=python-3.5.4-macosx10.9.pkg
-
local inst_path=downloads/python-3.5.4-macosx10.9.pkg
-
mkdir -p downloads
-
curl https://www.python.org/ftp/python/3.5.4/python-3.5.4-macosx10.9.pkg
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 162 100 162 0 0 1873 0 --:--:-- --:--:-- --:--:-- 1883 -
'[' pkg == dmg ']'
-
sudo installer -pkg downloads/python-3.5.4-macosx10.9.pkg -target /
installer: Error - the package path specified was invalid: 'downloads/python-3.5.4-macosx10.9.pkg'.
I understand multibuild is using macosx10.6.pkg for python 3.5.4, but libc++ are recommended to set a deployment target of macOS 10.9 or later. We changed 10.6 to 10.9, it worked well for the past 2 years but start to fail recently. Has multibuild changed the download place (it is now https://www.python.org/ftp/python/ ) or deleted python-3.5.4-macosx10.9.pkg recently?
Friendly ping? @matthew-brett
@robbuckley - any thoughts?
@matthew-brett and @robbuckley , is it possible to make this higher priority? Thanks to provide this build tool. I am from google protobuf team, this issue is blocking protobuf 3.11.2 release (All other languages have released 3.11.2, but python is blocked)
Upload python-3.5.4-macosx10.9.pkg to https://www.python.org/ftp/python/ can unblock this issue. Or is multibuild download the packages from other places before?
Sorry - this is our fault - we're assuming - in the newest code, that the 10.9 installer exists, but it does not, and presumably, never did.
Could you point to your build log to get the full traceback? And your CI files on Github?
And yes, we'll give it high priority - I will make sure to fix this before the end of the day tomorrow (GMT).
Hmm - I wonder - is it possible you are setting MB_PYTHON_OSX_VER to 10.9 for your Python 3.5 build matrix entry? If so, you need to remove that, it will cause multibuild to look for the wrong installer.
If you want to compile binaries targetting 10.9, for Pythons like 3.5, with 10.6 installers, you need to set MACOSX_DEPLOYMENT_TARGET=10.9 instead.
We have manually changed the OSX version to 10.9, I think that is the cause the the failure, but it was working well in the past:
https://github.com/protocolbuffers/protobuf/blob/master/python/setup.py#L197
We also have set MACOSX_DEPLOYMENT_TARGET=10.9 :
https://github.com/protocolbuffers/protobuf/blob/33af06edbb2940be7e9ddd6816e56884e7ecf8ad/kokoro/release/python/macos/build_artifacts.sh#L20
This is the full failure log:
https://source.cloud.google.com/results/invocations/724eefb2-3b97-46d2-a2cb-be15db08e54e/targets/protobuf%2Fgithub%2F3.11.x%2Fmacos%2Fpython_release%2Fpresubmit/log
Oh dear - I see the problem. Multibuild thinks itself to be in a one-shot environment, and it sets many environment variables to default values, thinking this will have no effect on anything other than the current run. In this case, because you first build Python 2.7, it sets MB_OSX_VER to a default value for Python 2.7 - of 10.9 - and then, when you run further builds in the same environment, it does not reset this value, and still has 10.9 as the architecture for Python 3.5, even though this does not work, for Python 3.5.
@robbuckley - what do you think - should we only set the default inside the functions, at function run-time?
Could you try this temporary fix?
# 10.6 installer
MB_OSX_VER=10.6
build_artifact_version 3.5
# 10.9 installer
MB_OSX_VER=10.9
build_artifact_version 2.7
build_artifact_version 3.6
build_artifact_version 3.7
Yes, this is the problem. But we still confused why it was working well in the past, we do released python3.5 for OSX 10.9 in the past, for example:
https://pypi.org/project/protobuf/3.11.1/#files
Use 10.6 for python3.5.4 will work well, but libc++ are recommended to set a deployment target of macOS 10.9 or later.
Do you know if it is possible that https://www.python.org/ftp/python/ has removed python-3.5.4-macosx10.9.pkg recently?
Oh, I checked the past log, it was downloading python-3.5.4-macosx10.6.pkg to build 10.9. I think reset MB_OSX_VER may fix it and is the correct way. Will try it and let you know the result. Thanks!
MACOSX_DEPLOYMENT_TARGET sets the SDK you use during compilation - the Python.org download only sets the default for this variable, if you have not set it explicitly. So you should be fine setting both MB_OSX_VER (to set the download file) and MACOSX_DEPLOYMENT_TARGET to set the SDK.
MB_OSX_VER does not work. We checked multibuild's code, looks like it should be MB_PYTHON_OSX_VER:
https://github.com/matthew-brett/multibuild/blob/2c84e1527704e42a02635b60f293f1b6c78462dc/osx_utils.sh#L408
Correct me if I am wrong. Will give another try
It works. Thank you matthew
protocolbuffers/protobuf#7038
hi @anandolee @matthew-brett sorry I only just caught up with this thread. I've fixed my email notifs so I should react quicker in future such cases :-/
@anandolee the build log link above doesnt work for me - i get an error "Uh oh! Something went wrong. ". But i think I can see the problem. I suggest a slightly better fix might be to add the following:
unset MB_PYTHON_OSX_VER
above this line
Then multibuild will set it automatically to the latest available version for the value of MB_PYTHON_VER
And then you should be able to remove both lines:
https://github.com/protocolbuffers/protobuf/blob/63cfdafacba6141717a2df97fc123dc0c14ba7c4/kokoro/release/python/macos/build_artifacts.sh#L53
and:
https://github.com/protocolbuffers/protobuf/blob/63cfdafacba6141717a2df97fc123dc0c14ba7c4/kokoro/release/python/macos/build_artifacts.sh#L60
@matthew-brett as you say multibuild assumes its started from a clean [ travis ] env each time -- source-ing common-utils.sh and travis-steps.sh doesnt reset the environment to a clean state. In particular variables like MB_PYTHON_VER and MB_PYTHON_OSX_VER persist and will change the behaviour of subsequent runs.
If you want to support this kind of use case - multiple build runs in the same env - I could have a go at it if you like?
@robbuckley - if you have interest - it would be good to think about how we could at least make it less dangerous to do this, perhaps without guarantees it will work for multiple runs in the same process. We could restrict ourselves to multiple runs on the same architecture.
sure @matthew-brett. I've opend #296 to capture ideas. It will be a little while before i can do anything more on this. happy new year!