Installation problem on MacOS 12.1 (m1max chip) with python 3.8.

[shijunVV]

Please read the README, particularly the known limitations section!

Describe the issue
When I tried to install pyreadr on my Mac with m1max chip (MacOS version 12.1, python version 3.8), I got an error saying "pyreadr/libs/librdata/src/rdata_read.c:29:10: fatal error: 'lzma.h' file not found" and the installation failed. "
(not sure if this is a useful information for investigating)

To Reproduce
I have tried all means I can come up with to install pyreadr.

1. pip install pyreadr
2. download the zip file from "https://pypi.org/project/pyreadr/" and install it using "pip install -e /path/to/unzipped/file" and get the same error as above.
3. downlaod the zip file and install with "python setup.py" and get an error "ValueError: 'pyreadr/librdata.pyx' doesn't match any files".
4. pip install -i https://pypi.anaconda.org/ofajardo/simple pyreadr and I got a error saying "The package index page being used does not have a proper HTML doctype declaration.".
5. download the whl file from "https://pypi.anaconda.org/ofajardo/simple/pyreadr" with name "pyreadr-0.4.4-cp310-cp310-macosx_10_9_universal2.whl" and use "pip install ***.whl" to install. An dismatch error popped up: "pyreadr-0.4.3-cp310-cp310-macosx_10_9_universal2.whl is not a supported wheel on this platform."

Expected behavior
Any solution to this installation problem or do I miss any dependency on my local?

Setup Information:
How did you install pyreadr? (pip, conda, directly from repo)
described above.

Platform (windows, macOS, linux, 32 or 64 bit)
macOS 12.1 (m1max chip)

Python Version
3.8

Python Distribution (System, plain python, Anaconda)
I have tried homebrew python, miniconda, and plain python.

Using Virtualenv or condaenv?
Virtualenv.

The entire error screenshot:
pic-1:

pic-2:

pic-3:

[ofajardo]

Please try installing lzma before installing pyreadr , something along these lines:
https://ports.macports.org/port/lzma/
Or brew install xz

[shijunVV]

Thanks for prompt response!

I have tried "brew install xz" as well as "brew reinstall xz" before installing pyreadr but the error related to lzma.h still popped up. Should I use macPort instead?

[ofajardo]

Hmm you can try macports.

It seems the issue is even if lzma is there python cannot find it. You could try to locate the lzma.h and add that directory to setup.py. look in line 38 you could copy that line below line 28 with the folder where lzma.h is. Maybe the absolute path maybe take a copy under pyreadr.
Instead of modifying setup.py You could also try to set the variable DYLD_LIBRARY_PATH with thefolder where lzma.h is before installing, in the same terminal.

Last thing you can try is in setup.py line 23 take out DHAVE_LZMA. Side effect is you wont be able to read files with lzma compression, but maybe that is not so disturbing.

[ofajardo]

And another very simple thing you can try is to install with conda instead of pip (see readme), as we have wheels for mac M1 on conda forge.
If you find a solution please share, it will be useful for others

[shijunVV]

Thanks for replying on weekend.

I still cannot fix the lzma.h problem with your suggestion, but the solution based on conda works!

As I need to install pyreadr in a virtual environment created by the "reticulate" package in R, the following is the steps:

1. install pyreadr through conda. (conda install -c conda-forge pyreadr)
2. copy all the pyreadr related folders from the site_package in conda to that in the virtual environment.
3. Done!

[ofajardo]

Ok happy to have helped. Just in case, You can also create conda environments with reticulate: https://rstudio.github.io/reticulate/reference/conda-tools.html

[rikardn]

I am one of the authors of pharmpy-core which is the package that @shijunVV was trying to install. We have a newly added dependency of pyreadr which caused the installation issue. Thanks @ofajardo for helping out and for the great pyreadr package!

I know that packaging for MacOS is difficult and it seems as if the wheel universal2 on pypi was not supported on the particular MacOS platform, but the one on conda worked. Do you know the reason for this?

[ofajardo]

I think it is because the universal does not support the M1 chip, conda-forge, where I build the conda packages, does support the arm M1 chip architechture (basically one has to build for this specific architechture, the universal does not work). I am exploring if multibuild, which is the tool I use to build the pypi wheels do support arm ... I am not sure if it already does, if not it should come soon I guess. I keep you posted.

[rikardn]

Thanks for the update. Let me know if I could help with testing.

[ofajardo]

No problem. Apparently it is already supported, but I can't get it to work, so I have created an issue here to get some help. Once I manage to produce the wheels it will definitely be useful if you could test it before uploading to pypi, once it is in pypi it cannot be changed, so it better be good. I let you know.

[ofajardo]

hi @rikardn Could you help me with something? It seems I managed to produce wheels for the mac arm, and I would like someone to test before uploading to pypi. The files are going to be deployed here (in a few minutes) with the suffix macosx_11_0_arm64.whl and versions for python 3.8, 3.9 and 3.10. It is possible that trying to install these raise an error about no appropiate wheel for platform. In such case, it would be interesting to know if renaming the wheel to *_macosx_12_0_arm64.whl as here would solve the issue.

In case you can test this, let me know what happens, It would be very helpful =)

[rikardn]

@shijunVV Is this something you could test?

[ofajardo]

I have uploaded the wheels to pypi so you can test to install directly from there

[ofajardo]

another user reports that the wheels on pypi are working fine, so closing this issue. Please report if there are still problems.