PDLPorters/pdl

PDL 2.083 build errors

perlboy-emeritus opened this issue · 14 comments

perlboy-emeritus commented

I've been trying for days to upgrade PDL from 2.021 to 2.083, and so far, nothing, including advice from ej_zg has resolved my issue, a segmentation fault in compilation of pdlcore.c, to wit:

cc -c  "-I/Users/carly/.cpanm/work/1684165713.4672/PDL-2.083/Basic/Core" -fno-common -DPERL_DARWIN -mmacosx-version-min=10.13 -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -I/opt/local/include -DPERL_USE_SAFE_PUTENV -O3   -DVERSION=\"2.083\" -DXS_VERSION=\"2.083\"  "-I/Users/carly/perl5/perlbrew/perls/perl-5.30.2/lib/5.30.2/darwin-2level/CORE"  -DPDL_PTHREAD  pdlcore.c
clang: error: unable to execute command: Segmentation fault: 11
clang: error: clang frontend command failed due to signal (use -v to see invocation)
Apple LLVM version 10.0.0 (clang-1000.10.44.4)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
clang: note: diagnostic msg: PLEASE submit a bug report to http://developer.apple.com/bugreporter/ and include the crash backtrace, preprocessed source, and associated run script.
clang: note: diagnostic msg

I am running Perl

$ perl -v
This is perl 5, version 30, subversion 2 (v5.30.2) built for darwin-2level
(with 1 registered patch, see perl -V for more detail)

on macOS 10.13.6. Per ej_zg "The segfault you show below is from a bug in that version of clang. Upgrade xcode and it should work.

libdf is part of HDF4, and the Alien package exists to wrap knowledge about that. PDL should install and work fine if HDF4 isn't present. "

I was running Xcode 9.2. I've upgraded command line tools twice, to 9.4.1 and then 10.1 but PDL refuses to build.

$ clang --version
Apple LLVM version 10.0.0 (clang-1000.10.44.4)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

When I upgrade Alien my package is up to date, but I am still missing libdf.a, and without that library HDF4 will just not build. I am now without options as to how to get this or any older versions (if I could even find them) of PDL to build. Please don't suggest upgrading macOS just to get the latest PDL. That would cost me days of work fetching, building and/or patching other essential software I must have. It ought not to be this difficult to do a simple version upgrade.

Any help would be much appreciated.

perlboy-emeritus commented

Since posting my build report I found HDF-4.2.16 for Apple clang 10,0.0 and gfortran 6.3.0 at https://portal.hdfgroup.org/display/support/HDF+4.2.16#files and was able to successfully execute the shell script HDF-4.2.16-Darwin.sh. This script seemed to execute without error and built a huge tree of files including libdf.a. The question now is, what do I do with this stuff to make it available to cpanm building PDL? In other words, where do I put this stuff, or a link to it OTHER THAN IN THE macOS SYSTEM DIRECTORIES. I depend on perlbrew to keep my private, non-Apple Perl implementations current. Apple owns and uses the Perl that came with the OS and I prefer to not touch it, if possible.

mohawk2 commented

In order to make this work, you would need to have Alien::HDF4 return suitable C compiler flags for the location you've got HDF installed in. A "proper" Alien package these days, built on top of Alien::Build, would make this very easy, using a "share install".

Something like an explanation of share vs system can be seen at the docs' most relevant section. If you were minded to make a full, "proper" Alien::Build-using Alien::HDF4 instead of the hacked-together thing I made, that document will help you do so, and would probably not be hard. It would also be a great contribution! If you are so minded, please put it together and make a pull-request on the relevant repository.

For your purposes, you could simply edit your installed Alien::HDF4::Install::Files module (locate it with perldoc -l Alien::HDF4::Install::Files) to return appropriate things.

Keeping up to date with the latest MacOS might cause you some short-term disruption, but would avoid nearly all of the problems you are experiencing, and would also close various security holes. I will not support an old, insecure, obsolete version of MacOS, nor of clang, but I will be pleased to accept patches that make those work.

perlboy-emeritus commented
PerlSAP commented

Interesting find re: the C standard macro length.

However, wouldn't a truly open-source-informed approach be to submit a patch/ pull request that fixes the macro in question? (I.e. a couple of appropriately parametrised sub-macros that get incorporated in the currently huge one ought to do it. Hint hint).

perlboy-emeritus commented
mohawk2 commented

I am sure it has taken less time for me to write and run this code than it took @perlboy-emeritus to type in his humble-brag life story (whose relevance to this issue I cannot fathom):

$/=undef;
$_=<>;
push @found, [$1,$2]
  while /
    ^
    [ \t]*
    \#[ \t]*
    define[ \t]+
    ([a-zA-Z][^\(\s]+)
      [ \t]*\(.*?\)[ \t]*
    (.*?[^\\]\n)
  /gmsx;
#use Data::Dumper; print Dumper \@found;
print "$_->[0]: ", length($_->[1]), "\n" for @found;

which when run on pdlcore.c gives:

GEN_PDL_BARF_OR_WARN_I_STDARG: 298
INNERLOOP_X: 1333
PDL_KLUDGE_COPY_X: 4884
PDL_SETAV_X: 3131
mohawk2 commented

And the deep, cross-platform refactoring of the relevant macro by removing some whitespace has now reduced it to 3957 characters, which took <15 mins.

mohawk2 commented

Now released as 2.084.

perlboy-emeritus commented
perlboy-emeritus commented
perlboy-emeritus commented
mohawk2 commented

A real software engineer would:

  • know my Perl expression does not modify anything
  • realise C macro length includes whitespace
  • know that CPP operates before any lexing stage, rendering the "whitespace token" points above irrelevant
  • know how to use git to see different versions of files
  • be capable of capturing exactly the before-and-after versions of any files they modified in building/installing a piece of software
  • strip any such changes down to the bare minimum to achieve the change between success and failure
  • be able to submit such changes as a pull request
perlboy-emeritus commented
perlboy-emeritus commented