pip installed refl1d on Windows requires Python38.dll
Closed this issue · 23 comments
When installing refl1d on Windows using pip (I'm using an Anaconda distribution), reflmodule.pyd is linked directly to Python38.dll. This is true even if the python version used for the pip installation is 3.7, which then breaks reflmodule, raising an "ImportError: DLL not found" exception.
Note that installing refl1d under Python 3.8 allows import of reflmodule, but if a specific version of Python is required to run refl1d, this should perhaps be a dependency (or perhaps relaxed -- is there any reason to not allow 3.7, for example?)
It shouldn't. Here's the link line:
2021-05-30T17:58:31.5250583Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29910\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\hostedtoolcache\windows\Python\3.8.10\x64\libs /LIBPATH:C:\hostedtoolcache\windows\Python\3.8.10\x64\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29910\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29910\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64" /EXPORT:PyInit_reflmodule build\temp.win-amd64-3.8\Release\refl1d\lib\reflmodule.obj build\temp.win-amd64-3.8\Release\refl1d\lib\methods.obj build\temp.win-amd64-3.8\Release\refl1d\lib\reflectivity.obj build\temp.win-amd64-3.8\Release\refl1d\lib\magnetic.obj build\temp.win-amd64-3.8\Release\refl1d\lib\contract_profile.obj build\temp.win-amd64-3.8\Release\refl1d\lib\convolve.obj build\temp.win-amd64-3.8\Release\refl1d\lib\convolve_sampled.obj /OUT:build\lib.win-amd64-3.8\refl1d\reflmodule.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\refl1d\lib\reflmodule.lib
Simplified so it's easier to see that there are not refs to a python dll:
link.exe /nologo ... /EXPORT:PyInit_reflmodule reflmodule.obj methods.obj ... /OUT:reflmodule.pyd /IMPLIB:reflmodule.lib
Or any python libs for that matter. That's a bit surprising. I guess they use header magic to do all the version-independent binding inline rather than calling out to functions to do so.
One possibility is the version on pypi is out of date, and we did the version independent change since last release (though I don't think so).
Can you please download Dependency Walker and check what the .pyd in the windows wheel file references? This should tell us whether the python version is hard-coded into the package.
Another possibility is that there is an indirect reference to a python dll, and it is picking the first one up on the library path regardless of where it is, while a different version is on your command path.
I originally used https://github.com/lucasg/Dependencies to find the Python38.dll dependency. I will check whether a fresh Python 3.7 environment has this problem too.
Confirming that this fails with a fresh Python 3.7 environment. After creating the new environment (Python 3.7.10), I simply ran "conda install matplotlib numpy scipy" and then "pip install refl1d". The result is:
Python 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 15:37:01) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
import refl1d
import refl1d.reflmodule
Traceback (most recent call last):
File "", line 1, in
ImportError: DLL load failed: The specified module could not be found.
I can see the python38.dll in reflmodule.pyd from the windows wheel downloaded from pypi. I still don't understand how it got there since there are no references on the link line. I also skimmed through the python imports but didn't find anything obvious that would lead to an implicit link to the dll.
It may be that the link line has changed since Feb. and the next release will fix the problem. I couldn't figure out how to retrieve the build artifacts from github actions to check.
I guess try the Py_LIMITED_API change and see if the problem fixes itself?
BTW, I was able to use "strings reflmodule.pyd | grep -i dll" to see that it was linked.
The question is whether the link line has changed since the release. Looking at the build logs I don't see how the python38.dll is getting linked in, but it is clearly there in the latest github release.
Options:
(0) require python 3.8,
(1) build a separate wheel for each python version,
(2) send length and data pointer rather than numpy vectors,
(3) convert c code to numba and/or cuda and/or opencl, or
(4) other?
(1) is what we did before, and it's pretty straightforward. We're only supporting a handful of versions (3.6-3.9 currently)
(3) numba is something to consider, pretty easy to start using. Maintaining CUDA/CL is a big job. Wasmer is not inherently parallel but many engines can be run in parallel, and it is not just python version independent but platform independent! (also just a pip install)
In the short term, @pkienzle - should I adjust the actions scripts so that we generate versioned wheels, and then push out a quick new version 0.8.15? The current version is pretty seriously broken because of this.
At minimum I think we should delete the wheel files because they are incorrectly labelled as abi3.
I have generated wheels for v0.8.14 on WIndows, py36 through py39 (on a NIST windows laptop,) and can upload to pypi anytime. I removed the broken "abi3" wheel from pypi already.
A v0.8.15 release is appropriate. The main new feature (assuming we unpin periodictable) is that we use energy dependent Gd for monochromatic sources. I think the bugs related to the change in periodictable interface have been fixed now (#122), both for polarized and unpolarized.
You did the build by hand? I only see 3.8 mentioned in the github action.
I'm not seeing a PR for this, or a branch.
I merged it after we discussed it... now I would say prematurely.
Windows wheels have been added to version 0.8.14 on pypi. The build script has been modified to make separate wheels for python 3.6 - 3.9 on Windows and 3.7 - 3.9 on MacOS, in preparation for the next release.
Thanks!
Release 0.8.15 will not need binary wheels. It will be built with a numba dependency, which does JIT compiling of the things that need it.