Issue with installing python routines

Question

Issue with installing python routines

Closed this issue 2 years ago · 26 comments

Hi, I am new to PRESTO and have been attempting to install it on my personal laptop. After completing the main installation, I attempted to install the python routines (as suggested in INSTALL.md) but ran into an error. When I tried to run pip install . in the directory I've cloned and installed Presto into, I got this error:

ERROR: Could not build wheels for presto which use PEP 517 and cannot be installed directly

I was able to work around this problem (I think) using a command I found on StackOverflow:

sudo pip3 install . --no-binary :all:

but when I run this, while the installation gets further than it originally did, it eventually terminates with this error:

/usr/bin/ld: cannot find -lpresto
    collect2: error: ld returned 1 exit status
    error: Command "x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/python/presto_src/presto_wrap.o -lpresto -lfftw3f -lm -o build/lib.linux-x86_64-3.6/_presto.cpython-36m-x86_64-linux-gnu.so -Wl,-rpath,/tmp/pip-qu_asrng-build/lib" failed with exit status 1
    
    ----------------------------------------
  Rolling back uninstall of presto
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-qu_asrng-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-nefi9yk0-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-qu_asrng-build/

I don't know what is causing the issue but would be very grateful for any suggestions you might have on how to work around it. I'm happy to share the full output from the attempted installation if it would be helpful. I'm using pip 21.2.4 and Python 3.8.5 on Ubuntu 18.04. Thanks very much for any advice you might be able to share on this, and for your time!

Answer 1 · 2021-08-16T19:32:08.000Z

Couple questions:

Are you using the system python, with system numpy etc? If so, you might need to run the "pip install ." as sudo.
For the cannot find "-lpresto" issue, do you have the PRESTO environment variable set and pointing to the git checkout directory? And does "libpresto.so" exist in $PRESTO/lib?

Answer 2 · 2021-08-16T19:59:18.000Z

Hi Scott, thanks so much for getting back to me on this so quickly!

I just realized something -- the information I gave in my original post is slightly wrong -- apparently when I run pip3 with sudo, it runs a different version than when I run it without sudo (think this may have something to do with my anaconda installation). So, with sudo, instead of the versions I originally gave, I've actually been using pip version 9.0.1 and python version 3.6.9. Perhaps I should try updating these to see if that makes a difference?
Yes, I did set the PRESTO environment variable to my git checkout directory. I checked in my presto/lib directory, and "libpresto.so" is there too.

Thanks again, really appreciate your time!
Ellie

Answer 3 · 2021-08-16T20:53:53.000Z

Ah, ok. So you are using anaconda. That's important. If you installed anaconda not as root, then you should be able to enable your anaconda environment, do a "which pip" and it should point to anaconda's version, and then you should be able to "pip install .". If that is still a problem, try doing "pip install --user ." The pip definitely has to be anaconda's pip, though!

Answer 4 · 2021-08-16T21:55:44.000Z

That's good to know, thanks! All right so (with conda activated), I did "which pip", and it did return the anaconda version, so I attempted to run both pip install . and pip install --user ., but both resulted in the following error:

ERROR: Failed building wheel for presto
Failed to build presto
ERROR: Could not build wheels for presto which use PEP 517 and cannot be installed directly

Not sure why this is happening? Can share more of the output if need be.

Answer 5 · 2021-08-16T21:58:42.000Z

Make sure you have installed and updated to the latest versions of pip, setuptools and wheel in you anaconda distro.

Answer 6 · 2021-08-16T22:08:20.000Z

I tried:

python -m pip install --upgrade pip
python -m pip install --upgrade setuptools
python -m pip install --upgrade wheel

with conda enabled, and each attempt resulted in a Requirement already satisfied message, so I guess that means everything is up to date? Thanks for your patience with helping me troubleshoot this.

Answer 7 · 2021-08-16T22:24:41.000Z

Hmmm. This is a strange one. Couple more things to try:
Can you tell me what you get when you run "pip --version"?
For the last one, when you try to upgrade pip/setuptools/wheel, what do you get with "which python"?
Finally, can you try adding "--no-cache-dir" to the "pip install ." command?

Answer 8 · 2021-08-16T23:25:22.000Z

pip --version gives: pip 21.2.4 from /home/ellie/src/anaconda3/lib/python3.8/site-packages/pip (python 3.8).

which python gives: /home/ellie/src/anaconda3/bin/python

Sadly it still doesn't work when I add "--no-cache-dir", for some reason.

Answer 9 · 2021-08-17T01:21:05.000Z

Hmmm. Well that stinks. What version of anaconda are you using? I'm using Ubuntu (although a much newer version than you, 21.04) and am using miniconda with Python3 and things work great. Maybe I can try using your version of anaconda and see if I see the same issues.

Answer 10 · 2021-08-17T01:35:08.000Z

Yeah it is weird! Sorry this is turning out to be so complicated. Here's the anaconda version info:

conda --version >> conda 4.9.2
anaconda --version >> anaconda Command line client (version 1.7.2)

Answer 11 · 2021-08-17T01:53:15.000Z

So strange! I just installed the newest anaconda python, did a conda init, did cd $PRESTO and then pip install . and it worked perfectly! I just don't understand this....

Are you getting any other messages at all with the errors when you try to do the pip install .?

Answer 12 · 2021-08-17T02:01:14.000Z

What version of wheel and setuptools do you have installed now?

Answer 13 · 2021-08-17T02:04:00.000Z

And, with conda enabled, what happens when you run pip install . --no-binary? (after making sure that it is the pip from anaconda)

Answer 14 · 2021-08-17T02:54:55.000Z

That's odd, I wonder what's going on. I hope I haven't made some strange mistake when doing the initial installation steps, apologies if so. I've attached a file with the full output that comes when I run pip install .. My wheel version is 0.37.0, and setuptools is version 57.4.0.

Also, when I try pip install . --no-binary, I get the following output, which seems not right to me:

Usage:   
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

--no-binary option requires 1 argument

pip_install_presto_output.txt

Answer 15 · 2021-08-17T13:05:53.000Z

Oops. My fault. I gave you the wrong order. Should be:
pip install --no-binary .
and if that still doesn't work, try the other one:
pip install --no-cache-dir .
Please post all of the output.

Both of those are online recommended ways of possibly dealing with this issue with Anaconda.

Answer 16 · 2021-08-17T13:16:38.000Z

No worries! I should have realized that from the output. All right so when I do pip install --no-binary, it gives this message:

ERROR: You must give at least one requirement to install (see "pip help install")

I've attached the full output from running pip install --no-cache-dir . here:
pip_install_nocache_presto_output.txt

Answer 17 · 2021-08-17T13:45:16.000Z

Note the "." at the end of the pip install --no-binary . command! That means "Do a pip install of the software found here (i.e. '.')"

Answer 18 · 2021-08-17T13:51:26.000Z

Ah-ha! That full output helped! Here is the bad line (I think):

  /home/ellie/src/anaconda3/compiler_compat/ld: /usr/local/pgplot/libpgplot.a(xwdriv.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC

Note that it is finding pgplot in /usr/local. Unless you have a really good reason for it, you don't want that. In fact, I'd recommend removing the /usr/local/pgplot directory entirely if you can.

You should make sure that you have the "pgplot5" Ubuntu package unstalled. And you should have the environment variable PGPLOT_DIR=/usr/lib/pgplot5 (i.e. "export PGPLOT_DIR=/usr/lib/pgplot5" in your .bashrc file).

Answer 19 · 2021-08-17T17:21:03.000Z

Yay! I did everything you suggested above and it worked!! Thank you so much for helping me troubleshoot this, and for your patience. So glad it is fixed now! :)

Answer 20 · 2021-08-17T17:52:24.000Z

Ok, sorry to ask one final thing -- but in running python tests/test_presto_python.py, I am having one error; when it gets to the point where it trys to test get_baryv, it fails with an error; here's the full output:

Warning:  '/home/ellie/src/presto/lib/fftw_wisdom.txt' is not up-to-date.
          You should run 'makewisdom'.  See $PRESTO/INSTALL.
Testing FFT stuff... success
Testing FFTW call... success
Warning:  '/home/ellie/src/presto/lib/fftw_wisdom.txt' is not up-to-date.
          You should run 'makewisdom'.  See $PRESTO/INSTALL.
Testing tablesixstepfft call... success
Testing reading infiles... success
Testing writing infiles... success
Note: If the following fails, edit $PRESTO/Makefile and follow
      the instructions about the CFLAGS USE_FFTW_MALLOC option
Testing allocation and freeing of memory... success
Testing psrparams and orbitparams stuff... success
Testing spectralpower and spectralphase... success
Testing vector shifting / rotation... success
Testing orbit integration stuff... success
Testing Fourier response generation... success
Testing angle functions... success

Error in chkfopen(): No such file or directory
   path = 'resid2.tmp'

If I comment out that particular test, everything else works. (Also, not sure what the makewisdom warnings are about, when I run it, it says everything's up to date). I tried to poke around a bit in the python code to see what might be going on but couldn't find where the get_baryv() function is defined, so not quite sure what's happening. Anyway sorry to bother you again and if you don't have time to look at this, no worries. Thank you so much! :)

Answer 21 · 2021-08-17T17:58:06.000Z

Do you have a working version of TEMPO installed?

Answer 22 · 2021-08-17T17:59:28.000Z

The testing error might be because TEMPO is either not installed correctly or misbehaving. Are you able to run "tempo" at all?

Also, just checking, once you got rid of the the "rogue" pgplot install, were you able to install just using "pip install ."?

(just saw that @paulray also posted about TEMPO)

Answer 23 · 2021-08-17T18:04:14.000Z

I suspect you're right that TEMPO is not installed correctly -- and actually, I think I might know where the problem came from, for both it and PGPLOT. I just tried to install psrchive before installing presto, and the way the installation instructions recommended installing TEMPO and PGPLOT seemed odd to me. But then again, I installed TEMPO2 for psrchive, so maybe that's not the problem.

In any case, when I tried to just run the command "tempo", this is what happened:

more: stat of /usr/local/bin/tempo/tempo.hlp failed: Not a directory

And yes, after removing the rogue pgplot, "pip install ." worked perfectly :)

Answer 24 · 2021-08-17T18:34:36.000Z

OK. I think you are really close. Make sure you grab the TEMPO source from here:
http://tempo.sourceforge.net/

Then in the directory, doing "./prepare ; ./configure --prefix=/home/ellie ; make ; make install" should do it. Then just make sure that your TEMPO environment variable is pointing to the source directory for TEMPO.

Answer 25 · 2021-08-17T18:58:50.000Z

Awesome, that worked, how exciting! I should be good to go now. Thank you so much!! :)

Answer 26 · 2021-08-17T19:10:47.000Z

Great! Good luck!