bingos/devel-patchperl

Tests for h2ph.t, op/numconvert.t, and op/range.t fails when trying to install 5.14.4 (using gcc 4.9)

hakonhagland opened this issue · 11 comments

hakonhagland commented

I am trying to install an earlier perl version (5.14.4) using
perlbrew, Ubuntu 14.10, and perl version 5.20.1. Installation failed at running "make test"..

I figured this was not a problem with perlbrew, and downloaded the source myself from www.cpan.org :

$ wget http://www.cpan.org/src/5.0/perl-5.22.0.tar.gz
$ tar -xzf perl-5.14.4.tar.gz
$ cd perl-5.14.4
$ ./Configure -des -Dprefix=$HOME/localperl
$ make
$ make test

and the same tests failed. Here is the output from running /.myconfig in the installation folder:

$ ./myconfig
Summary of my perl5 (revision 5 version 14 subversion 4) configuration:

  Platform:
    osname=linux, osvers=3.16.0-44-generic, archname=x86_64-linux
    uname='linux hakon-thinkpad-edge-e540-2 3.16.0-44-generic #59-ubuntu smp tue jul 7 02:07:39 utc 2015 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Dprefix=/home/hakon/localperl'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.9.1', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib
    libs=-lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.19'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'

and here are the failed tests:

# Failed at ../lib/h2ph.t line 49
#      got 'Illegal declaration of subroutine main::__has_include at _h2ph_pre.ph line 488.
# '
# expected /(?^:syntax OK$)/
# Failed at ../lib/h2ph.t line 56
#      got "Illegal character in prototype for main::__has_include : STR at _h2ph_pre.ph line 488.\nCompilation failed in require at lib/h2ph.pht line 1.\nCompilation failed in require at -e line 1.\n"
# expected ""
../lib/h2ph.t ..................................................... 
Failed 2/6 subtests 


# '-9223372036854775808' ne '-9.22337203685478e+18',    9223372036854775808 => I - n P vs I - P
# Failed at op/numconvert.t line 247
# '-9223372036854775808' ne '-9.22337203685478e+18',    -9223372036854775808    => N - n P vs N - P
# Failed at op/numconvert.t line 247
# '-9223372036854775808' ne '-9.22337203685478e+18',    -9223372036854775808    => P - n P vs P - P
# Failed at op/numconvert.t line 247
op/numconvert.t ................................................... 
Failed 12/1444 subtests 


# Failed at op/range.t line 311
#      got undef
# expected "-9223372036854775807"
# Failed at op/range.t line 312
#      got undef
# expected "-9223372036854775804"
# Failed at op/range.t line 330
op/range.t ........................................................ 
Failed 25/141 subtests

Note, I first reported this issue against perlbug@perl.org :

https://rt.perl.org/Public/Bug/Display.html?id=126820

n1vux commented

Seeing the same thing updating my Perlbrews on new ubuntu, needed because new desktop has new machine arch (Core i7 x86_64-linux instead of i686-linux previously), doing this for Hacktoberfest to test if a module still builds on claimed min Perl requirement etc.

There are two separate issues here.

1 The op/numconvert.t (and likely op/range.t too) seems to be related to a GCC4.9 optimization change reported to perl core as #121505 by Fedora project interacting badly with sv.c being willing to do an undefined arithmetic operation (decrement min int, increment max int, etc) and Perl core test suite verifying that it is done correctly. GCC figures it can optimize an undefined operation anyway it likes. It was deemed 5.20.0 issue then. I do not immediately spot the fix in either op/numconvert.t or sv.c in 5.20.x, however. This was documented in perl5200delta.pod

I suspected the fix in 5.20 and forward actually a GCC flag when building with Gcc4.9 or higher to suppress the optimization, rather than a fix to code or test; and we need to include that when using Perlbrew to build older Perls whose cflags.SH don't generate the flag. I'm guessing ccflags=-fwrapv and same in cppflags from context in Fedora report above and comparing config.sh in perlbrew/build/*/.

If PatchPerl can patch Configure to add -frwrapv logic (for Gcc4.3+ unless contradictory option requested) on pre-5.20.x older Perls that don't have it, that would solve it for Perlbrew. (I have NOT checked with 5.19.x added it.)

Workaround would appear to be passing Configure argumement -A ccflags=-fwrapv to perlbrew , either on commandline or in ENV as PERLBREW_CONFIGURE_FLAGS='-de -A ccflags=-fwrapv' ? Well, that cures the op/numconvert.t fails, but doesn't cure ../cpan/CGI/t/tmpdir.t TODO passed: 3-9 which sounds harmless (passed tests that are marked TODO) and ../lib/h2ph.t Failed tests: 5-6

2 The above workaround (and updating perl-patch) do NOT help the ../lib/h2ph.t issue with e.g. perl-5.16.3 :

# Failed test 6 - output free of warnings at ../lib/h2ph.t line 56
#      got "Bareword found where operator expected at _h2ph_pre.ph line 210, near \"7fff\"\nCompilation failed in require at lib/h2ph.pht line 1.\nCompilation failed in require at -e line 1.\n"
# expected ""
../lib/h2ph.t .. Failed 2/6 subtests

This could mean an -A cppflags= but the only difference I see with recent perls is adding -strong on -fstack-protector-strong instead of -fstack-protector which doesn't sound like formatting of hex constants??

(Bugreport Perl 124062 has hints on debugging h2ph.t problems.)

mackdoughe commented

To extend from the previous comment, I've been trying to use perlbrew to install older versions of Perl (specifically v5.18.2) and have run in to the same issues. As @n1vux said, the errors that arise from op/numconvert.t and op/range.t come from a change in GCC and was mitigated in later versions of Perl with a compiler flag (see Perl 121505).

The ../lib/h2ph.t issue is unrelated but still stems from newer versions of GCC. The discussion, and ultimate patches, for this issue are in Bugreport Perl 123784.

From looking at PatchPerl source, it looks like the first patch mentioned in Bugreport 123784 for Errno was already implemented and the affected tests succeed. However, the remaining two patches that fix ../lib/h2ph.t were not. See the respective patches here and here.

I was successfully able to build an older version of Perl (5.18.2) on Ubuntu 18.04 from source today using the following methods:

  1. $ wget http://www.cpan.org/src/5.0/perl-5.18.2.tar.gz
    $ tar -xzf perl-5.18.2.tar.gz
    $ cd perl-5.18.2
  2. Manually apply the Errno patch that PatchPerl already handles
  3. Manually apply the ../lib/h2ph.t patch
  4. Manually apply the utils/h2ph.PL patch
  5. $ ./Configure -des -Dprefix=$HOME/localperl -A ccflags=-fwrapv
    $ make
    $ make test

So I think to get PerlPatch (and ultimately perlbrew) working again for these older versions of Perl with the new GCC, the ../lib/h2ph.t patch, utils/h2ph.PL patch, and -fwrapv flag mentioned by @n1vux need to be added.

davidcantrelluk2 commented

I had this problem with 5.8.9 and can confirm that if after running patchperl I build perl thus all the tests pass:

./Configure -de -Duse64bitall -Dusethreads -Dprefix=... -A ccflags=-fwrapv && make && make test

Update: 5.10.1 also

DrHyde commented
  1. Manually apply the ../lib/h2ph.t patch
  2. Manually apply the utils/h2ph.PL patch

that works for 5.18, but not when I tried it on 5.14.4 and 5.16.3. They both fail along these lines:

david@pigsty:~/cpantesting/perl-5.14.4-src$ ./perl -Ilib lib/h2ph.t                                                   
1..6                                                                                                                  
ok 1 - output is free of warnings                                                                                     
ok 2 - ../utils/h2ph runs successfully                                                                                
ok 3 - generated file has expected contents                                                                           
ok 4 - output compiles                                                                                                
not ok 5 - preamble compiles                                                                                          
# Failed at lib/h2ph.t line 49                                                                                        
#      got 'Illegal declaration of subroutine main::__has_include at _h2ph_pre.ph line 527.                           
# '                                                                                                                   
# expected /(?^:syntax OK$)/                                                                                          
not ok 6 - output free of warnings                                                                                    
# Failed at lib/h2ph.t line 56                                                                                        
#      got "Illegal character in prototype for main::__has_include : STR at _h2ph_pre.ph line 527.\nCompilation failed in require at lib/h2ph.pht line 1.\nCompilation failed in require at -e line 1.\n"                                           
# expected ""

A reasonable failure, but after a bit of digging to see what's in _h2ph_pre.ph I'm happy that it's a bug that the various patches have introduced into the tests and not in perl itself.

bingos commented

I have released 1.58 to CPAN which should address these issues.

hakonhagland commented

Thanks for working on this. But there still seems to be problems.. I am on Ubuntu 19.04, gcc version 8.3.0:

$ perlbrew install perl-5.10.1
[...]
op/quotemeta.t ................................................... ok
op/rand.t ........................................................ ok
# Failed at op/range.t line 289
# Failed at op/range.t line 290
#      got undef
# expected '-9223372036854775807'
# Failed at op/range.t line 291
#      got undef
# expected '-9223372036854775797'
# Failed at op/range.t line 289
# Failed at op/range.t line 290
#      got undef
# expected '-9223372036854775806'
[...]

op/range.t ....................................................... 
Failed 25/135 subtests 
op/re.t .......................................................... ok
[...]

See also this issue on SO.

hakonhagland commented

Sorry my bad, I had forgot to update perlbrew. This seems to work fine now! Thanks!

$ perlbrew install-patchperl
$ perlbrew install perl-5.10.1
[...]
p/quotemeta.t ................................................... ok
op/rand.t ........................................................ ok
op/range.t ....................................................... ok
op/re.t .......................................................... ok
[...]
alvations commented

When installing perl-5.10.1, I'm getting some failure to download

$ perlbrew install perl-5.10.1
Fetching perl 5.10.1 as /home/ltan/perl5/perlbrew/dists/perl-5.10.1.tar.bz2
Download http://www.cpan.org/authors/id/D/DA/DAPM/perl-5.10.1.tar.bz2 to /home/ltan/perl5/perlbrew/dists/perl-5.10.1.tar.bz2
ERROR: Failed to download http://www.cpan.org/authors/id/D/DA/DAPM/perl-5.10.1.tar.bz2

But wget works fine:

$ wget http://www.cpan.org/authors/id/D/DA/DAPM/perl-5.10.1.tar.bz2
--2019-11-27 15:58:23--  http://www.cpan.org/authors/id/D/DA/DAPM/perl-5.10.1.tar.bz2
Resolving www.cpan.org (www.cpan.org)... 151.101.2.217, 151.101.66.217, 151.101.130.217, ...
Connecting to www.cpan.org (www.cpan.org)|151.101.2.217|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11608061 (11M) [application/x-bzip2]
Saving to: ‘perl-5.10.1.tar.bz2’

perl-5.10.1.tar.bz2                      100%[==============>]  11.07M  5.75MB/s    in 1.9s    

2019-11-27 15:58:25 (5.75 MB/s) - ‘perl-5.10.1.tar.bz2’ saved [11608061/11608061]

And file:// didn't seem to work too =(

$ perlbrew install file://perl-5.10.1.tar.bz2
Installing perl-5.10.1 from local archive file://perl-5.10.1.tar.bz2
tar (child): /perl-5.10.1.tar.bz2: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
Failed to extract /perl-5.10.1.tar.bz2 at /usr/share/perl5/App/perlbrew.pm line 1275.
alvations commented

Whoops, sorry my mistake, should have followed the instructions on https://github.com/gugod/App-perlbrew

Getting perl-5.19.9 works =)

azrle commented

It doesn't work for perl-5.16.3 on Ubuntu 18.04 for the same errors of h2ph stated in the original post.

The following patch may be required:

--- Configure
+++ Configure
@@ -21959,8 +21959,9 @@
     tHdrH=_tmpHdr
     rm -f $tHdrH'.h' $tHdrH
     touch $tHdrH'.h'
+    # Filter out macro arguments, such as Linux's __INT8_C(c)
     if $cpp -dM $tHdrH'.h' > $tHdrH'_cppsym.h' && [ -s $tHdrH'_cppsym.h' ]; then
-       sed 's/#define[\ \  ]*//;s/[\ \     ].*$//' <$tHdrH'_cppsym.h' >$tHdrH'_cppsym.real'
+       sed -e 's/#define[\ \  ]*//;s/[\ \     ].*$//' -e 's/(.*//' <$tHdrH'_cppsym.h' >$tHdrH'_cppsym.real'
        if [ -s $tHdrH'_cppsym.real' ]; then
          cat $tHdrH'_cppsym.real' Cppsym.know | sort | uniq | ./Cppsym | sort | uniq > Cppsym.true
        fi

The reason why we are seeing

got 'Illegal declaration of subroutine main::__has_include at _h2ph_pre.ph line 488.

is that the symbols generated are including

$ echo | cpp -dM | grep has_include
#define __has_include(STR) __has_include__(STR)
#define __has_include_next(STR) __has_include_next__(STR)

which will be translated to

$ ./perl -I./lib ./utils/h2ph -d. -Q lib/h2ph.h
Skipping 'lib/h2ph.h':  not a file or directory

$ grep has_include _h2ph_pre.ph
unless (defined &__has_include(STR)) { sub __has_include(STR)() { "__has_include__\(STR\)" } }
unless (defined &__has_include_next(STR)) { sub __has_include_next(STR)() { "__has_include_next__\(STR\)" } }

in _h2ph_pre.ph file and it certainly leads to syntax error.

$ ./perl -I./lib -c _h2ph_pre.ph
Illegal declaration of subroutine main::__has_include at _h2ph_pre.ph line 689.

The above patch will fix this issue.

This fix is introduced from 5.17.1 by this commit Perl/perl5@6f87f40. We may need to patch versions before it.

hakonhagland commented

got 'Illegal declaration of subroutine main::__has_include at _h2ph_pre.ph"

See also this issue on stackoverflow.com