gudrunhe/secdec

Compilation never finishes

vsht opened this issue · 13 comments

vsht commented

Hi,

I tried to use pySecDec to check the following 1-scale eikonal integral

int
int2

with mg^2=1 and n^2=1. The generation step finishes without any issues but when compiling the code
it seems like the compilation itself never finishes and just prolongs itself indefinitely. It seems that
here FORM permanently runs into workspace overflows and so pySecDec keeps on increasing it.

I've already used pySecDec for similar integrals with more propagators and loops and it always worked flawlessly,
so I'm a bit surprised that this particular integral is causing so much troubles. Here is my config file

#! /usr/bin/env python
from pySecDec.loop_integral import loop_package
import pySecDec as psd

li = psd.loop_integral.LoopIntegralFromPropagators(
['p1**2-mg**2','p3**2-mg**2','2*n*p1','2*n*(p1+p3)'],
loop_momenta = ['p1','p3'],
replacement_rules = [ ('n**2','1'), ('mg**2','1') ]
)

loop_package(
name = 'myint',
additional_prefactor = 'exp(2*EulerGamma*eps)',
loop_integral = li,
real_parameters = [],
requested_order = 2,
form_optimization_level = 2,
decomposition_method = 'geometric',
)

I'm using the newest version of pySecDec installed via pip, but this particular issue was present already on 1.4.5.
My Python version is 3.9.7, and the GCC version is 10.3.1 (Fedora linux). I'm doing the evaluation on a
Ryzen 4750U (8c/16t) and 32 GB RAM, so the available resources are not an issue here.

The result returned by FIESTA (nonpublic beta) reads

ep^2 (0.274852` pm43+47.718132`)+ep (0.041496` pm42+6.063054`)+0.018345` pm41+4.933928`

and is also in agreement with my analytic result.

magv commented

The reason it takes so long is that the expansion starts at eps^-4, so with requested_order = 2 that's 7 orders of expansion. The expressions grow, and it takes some effort for FORM to optimize them; the final WorkSpace is 1350000000.

Anyway, the compilation does finish for me; it just takes 2+ hours. The integration is surprisingly fast; here's the result:

amplitude0 = 
    + ((0,0) +/- (0,0))*eps^-4
    + ((-9.39989e-14,-3.05138e-13) +/- (1.02237e-12,6.77283e-13))*eps^-3
    + ((-3.75754e-10,3.54838e-10) +/- (1.98196e-10,4.34774e-10))*eps^-2
    + ((-5.55113e-08,-8.16597e-08) +/- (2.97847e-08,4.16706e-08))*eps^-1
    + ((4.9348,-1.18909e-07) +/- (4.33455e-07,4.31441e-07))
    + ((6.05703,4.80039e-06) +/- (5.8618e-06,4.50586e-06))*eps
    + ((47.809,-0.000468736) +/- (0.000534307,0.000408118))*eps^2
    + O(eps^3)
magv commented

And if you'll set contour_deformation=False the whole thing is over in minutes :) Try it.

vsht commented

And if you'll set contour_deformation=False the whole thing is over in minutes :) Try it.

Hmm, with contour_deformation=False in my case it doesn't seem to compile at all

╰─ make -j8 -C myint      
make: Entering directory '/home/vs/Downloads/TestInt/myint'
g++ -c -std=c++14 -O2 -I. -I/home/vs/Downloads/TestInt/myint/ -I"/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/include" -fPIC pylink/pylink.cpp -o pylink/pylink.o
g++ -c -std=c++14 -O2 -I. -I/home/vs/Downloads/TestInt/myint/ -I"/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/include" -fPIC src/amplitude.cpp -o src/amplitude.o
g++ -c -std=c++14 -O2 -I. -I/home/vs/Downloads/TestInt/myint/ -I"/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/include" -fPIC src/myint_integral_weighted_integral.cpp -o src/myint_integral_weighted_integral.o
make -C myint_integral/ libmyint_integral.a
g++ -c -std=c++14 -O2 -I. -I/home/vs/Downloads/TestInt/myint/ -I"/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/include" -fPIC pylink/qmc_template_instantiations_0.cpp -o pylink/qmc_template_instantiations_0.o
make[1]: Entering directory '/home/vs/Downloads/TestInt/myint/myint_integral'
cd codegen && python3 formwrapper "/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/bin/tform" -M -w2 -D optimizationLevel=2 -D sectorID=1 write_integrand.frm
cd codegen && python3 formwrapper "/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/bin/tform" -M -w2 -D optimizationLevel=2 -D sectorID=3 write_integrand.frm
cd codegen && python3 formwrapper "/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/bin/tform" -M -w2 -D optimizationLevel=2 -D sectorID=2 write_integrand.frm
cd codegen && python3 formwrapper "/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/bin/tform" -M -w2 -D optimizationLevel=2 -D sectorID=4 write_integrand.frm
TFORM 4.2.1 (Nov 21 2018, v4.2.1) 64-bits 2 workers  Run: Thu Sep  2 21:23:13 2021
    #-
  0.26 sec + 0.27 sec: 0.54 sec out of 0.40 sec
python3 codegen/move \
	codegen/sector_3.cpp \
	codegen/*sector_3_*.[ch]pp \
	src/
touch codegen/sector3.done
cd codegen && python3 formwrapper "/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/bin/tform" -M -w2 -D optimizationLevel=2 -D sectorID=5 write_integrand.frm
TFORM 4.2.1 (Nov 21 2018, v4.2.1) 64-bits 2 workers  Run: Thu Sep  2 21:23:13 2021
    #-
  0.35 sec + 0.36 sec: 0.71 sec out of 0.55 sec
python3 codegen/move \
	codegen/sector_2.cpp \
	codegen/*sector_2_*.[ch]pp \
	src/
touch codegen/sector2.done
cd codegen && python3 formwrapper "/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/bin/tform" -M -w2 -D optimizationLevel=2 -D sectorID=6 write_integrand.frm
g++ -c -O2 -std=c++14 -I. -I/home/vs/Downloads/TestInt/myint/myint_integral/codegen/ -I"/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/include" -fPIC src/integrands.cpp -o src/integrands.o
TFORM 4.2.1 (Nov 21 2018, v4.2.1) 64-bits 2 workers  Run: Thu Sep  2 21:23:14 2021
    #-
  0.87 sec + 1.24 sec: 2.11 sec out of 1.47 sec
python3 codegen/move \
	codegen/sector_6.cpp \
	codegen/*sector_6_*.[ch]pp \
	src/
touch codegen/sector6.done
g++ -c -O2 -std=c++14 -I. -I/home/vs/Downloads/TestInt/myint/myint_integral/codegen/ -I"/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/include" -fPIC src/pole_structures.cpp -o src/pole_structures.o
g++ -c -O2 -std=c++14 -I. -I/home/vs/Downloads/TestInt/myint/myint_integral/codegen/ -I"/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/include" -fPIC src/prefactor.cpp -o src/prefactor.o
TFORM 4.2.1 (Nov 21 2018, v4.2.1) 64-bits 2 workers  Run: Thu Sep  2 21:23:13 2021
    #-
free(): invalid pointer
make[1]: *** [Makefile:41: codegen/sector1.done] Error 250
make[1]: *** Waiting for unfinished jobs....
TFORM 4.2.1 (Nov 21 2018, v4.2.1) 64-bits 2 workers  Run: Thu Sep  2 21:23:14 2021
    #-
  1.67 sec + 2.93 sec: 4.61 sec out of 3.61 sec
python3 codegen/move \
	codegen/sector_5.cpp \
	codegen/*sector_5_*.[ch]pp \
	src/
touch codegen/sector5.done
TFORM 4.2.1 (Nov 21 2018, v4.2.1) 64-bits 2 workers  Run: Thu Sep  2 21:23:13 2021
    #-
  3.91 sec + 11.35 sec: 15.26 sec out of 9.77 sec
python3 codegen/move \
	codegen/sector_4.cpp \
	codegen/*sector_4_*.[ch]pp \
	src/
touch codegen/sector4.done
make[1]: Leaving directory '/home/vs/Downloads/TestInt/myint/myint_integral'
make: *** [Makefile:17: myint_integral/libmyint_integral.a] Error 2
make: *** Waiting for unfinished jobs....
make: Leaving directory '/home/vs/Downloads/TestInt/myint'

Am I doing something wrong?

magv commented

When FORM crashes with free(): invalid pointer that's a FORM problem :) I'll look into it; for the moment you can build with env FORM=your-other-form-binary make.

vsht commented

Many thanks for your help!

Indeed, using tform freshly compiled from the master branch makes the compilation finish successfully. It seems
like the fixes that were added to the expression optimizer in July and August 2020 are crucial here. Perhaps it would
be worth to update the FORM binary shipped with pySecDec ;)

Unfortunately, I'm now having an issue with the actual integration (I was using 1.4.5 before switching to the latest stable release)

╰─ python integrate_int.py                                              
Traceback (most recent call last):
  File "/home/vs/Downloads/TestInt/integrate_int.py", line 5, in <module>
    myint_3L = IntegralLibrary('myint/myint_pylink.so')
  File "/home/vs/.local/lib/python3.9/site-packages/pySecDec/integral_interface.py", line 1014, in __init__
    c_lib = self.c_lib = CDLL(shared_object_path)
  File "/usr/lib64/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: myint/myint_pylink.so: undefined symbol: _ZN5GiNaC18get_default_readerB5cxx11Ev

My ginac version seems to be fine, though: ginac-1.7.9-5.fc33

magv commented

That might be the problem, yeah. I'll probably need to update FORM; .

As for the GiNaC problem, I can't seem to reproduce it. What does ldd myint/myint_pylink.so say? Also try to cd myint && make ./integrate_myint && ./integrate_myint.

(Your system GiNaC version doesn't matter; pySecDec ships with its own copy).

vsht commented

Here's the output

% ldd myint/myint_pylink.so
	linux-vdso.so.1 (0x00007ffd2f170000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fd994eda000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fd994cf2000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd994cd7000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd994cb5000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fd994aea000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd995645000)

% cd myint && make ./integrate_myint && ./integrate_myint.
g++ -c -std=c++14 -O2 -I. -I/home/vs/Downloads/TestInt/myint/ -I"/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/include" -fPIC integrate_myint.cpp -o integrate_myint.o
ar -c -q libmyint.a src/amplitude.o src/myint_integral_weighted_integral.o
dir=$(mktemp -d) && \
cd $dir && \
ar -x "/home/vs/Downloads/TestInt/myint/myint_integral/libmyint_integral.a" && ar -q "/home/vs/Downloads/TestInt/myint/libmyint.a" *.o && rm -f *.o &&  \
rm -rf $dir
g++ -o integrate_myint integrate_myint.o libmyint.a -L"/home/vs/.local/lib/python3.9/site-packages/pySecDecContrib/lib" -pthread -lgsl -lgslcblas -lcuba -lm -lginac -lcln -lm
/usr/bin/ld: libmyint.a(myint_integral_weighted_integral.o): in function `secdecutil::ginac::ex_to_nested_series<secdecutil::Series<GiNaC::ex> >::convert(GiNaC::ex, std::vector<GiNaC::symbol, std::allocator<GiNaC::symbol> > const&, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, unsigned long)':
myint_integral_weighted_integral.cpp:(.text._ZN10secdecutil5ginac19ex_to_nested_seriesINS_6SeriesIN5GiNaC2exEEEE7convertES4_RKSt6vectorINS3_6symbolESaIS8_EERKS7_IiSaIiEESG_m[_ZN10secdecutil5ginac19ex_to_nested_seriesINS_6SeriesIN5GiNaC2exEEEE7convertES4_RKSt6vectorINS3_6symbolESaIS8_EERKS7_IiSaIiEESG_m]+0x314): undefined reference to `GiNaC::symbol::get_name[abi:cxx11]() const'
/usr/bin/ld: libmyint.a(myint_integral_weighted_integral.o): in function `secdecutil::Series<std::complex<double> > secdecutil::ginac::read_coefficient<secdecutil::Series, double, std::complex<double> >(std::istream&, std::vector<int, std::allocator<int> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<double, std::allocator<double> > const&, std::vector<std::complex<double>, std::allocator<std::complex<double> > > const&)':
myint_integral_weighted_integral.cpp:(.text._ZN10secdecutil5ginac16read_coefficientINS_6SeriesEdSt7complexIdEEET_IT1_ERSiRKSt6vectorIiSaIiEERKS9_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISJ_EESN_SN_RKS9_IT0_SaISO_EERKS9_IS6_SaIS6_EE[_ZN10secdecutil5ginac16read_coefficientINS_6SeriesEdSt7complexIdEEET_IT1_ERSiRKSt6vectorIiSaIiEERKS9_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISJ_EESN_SN_RKS9_IT0_SaISO_EERKS9_IS6_SaIS6_EE]+0x238): undefined reference to `GiNaC::symbol::symbol(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: myint_integral_weighted_integral.cpp:(.text._ZN10secdecutil5ginac16read_coefficientINS_6SeriesEdSt7complexIdEEET_IT1_ERSiRKSt6vectorIiSaIiEERKS9_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISJ_EESN_SN_RKS9_IT0_SaISO_EERKS9_IS6_SaIS6_EE[_ZN10secdecutil5ginac16read_coefficientINS_6SeriesEdSt7complexIdEEET_IT1_ERSiRKSt6vectorIiSaIiEERKS9_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISJ_EESN_SN_RKS9_IT0_SaISO_EERKS9_IS6_SaIS6_EE]+0x1028): undefined reference to `GiNaC::get_default_reader[abi:cxx11]()'
/usr/bin/ld: myint_integral_weighted_integral.cpp:(.text._ZN10secdecutil5ginac16read_coefficientINS_6SeriesEdSt7complexIdEEET_IT1_ERSiRKSt6vectorIiSaIiEERKS9_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISJ_EESN_SN_RKS9_IT0_SaISO_EERKS9_IS6_SaIS6_EE[_ZN10secdecutil5ginac16read_coefficientINS_6SeriesEdSt7complexIdEEET_IT1_ERSiRKSt6vectorIiSaIiEERKS9_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISJ_EESN_SN_RKS9_IT0_SaISO_EERKS9_IS6_SaIS6_EE]+0x104f): undefined reference to `GiNaC::parser::parser(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, GiNaC::ex, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, GiNaC::ex> > > const&, bool, std::map<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long>, GiNaC::ex (*)(std::vector<GiNaC::ex, std::allocator<GiNaC::ex> > const&), GiNaC::PrototypeLess, std::allocator<std::pair<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long> const, GiNaC::ex (*)(std::vector<GiNaC::ex, std::allocator<GiNaC::ex> > const&)> > > const&)'
/usr/bin/ld: myint_integral_weighted_integral.cpp:(.text._ZN10secdecutil5ginac16read_coefficientINS_6SeriesEdSt7complexIdEEET_IT1_ERSiRKSt6vectorIiSaIiEERKS9_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISJ_EESN_SN_RKS9_IT0_SaISO_EERKS9_IS6_SaIS6_EE[_ZN10secdecutil5ginac16read_coefficientINS_6SeriesEdSt7complexIdEEET_IT1_ERSiRKSt6vectorIiSaIiEERKS9_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISJ_EESN_SN_RKS9_IT0_SaISO_EERKS9_IS6_SaIS6_EE]+0x106f): undefined reference to `GiNaC::parser::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
collect2: error: ld returned 1 exit status
make: *** [Makefile:39: integrate_myint] Error 1
magv commented

No idea. Some strange compiler incompatibility with the prebuilt packages maybe? If that's the case, then git clone https://github.com/gudrunhe/secdec, cd secdec, make, and export PYTHONPATH=$PWD, and it should work... This is the sort of workflow that we wanted to avoid for our users; maybe try pip3 install --force-reinstall --no-binary :all: pySecDec instead: it does the same thing, but is easier to type.

vsht commented

Ok, issuing

python3 -m pip install . --user

from within the master branch I got everything working. Well, I'm already sort of used to these issues since Fedora is always
a bit capricious and no apparently one tests against it upstream.

As for the integral itself, turning contour_deformation off indeed fixes the issue I observe, but it is not such a nice
workaround if one has many integrals that must be evaluated numerically: Looking at each integral and figuring out
whether it can have an imaginary part or not, it something one would rather like to avoid.

Anyway, thanks a lot for your help. Should I close the issue now, or would you prefer to keep it open as a reminder for possible improvements?

magv commented

The ginac linking problem is definitely something I'd like to fix, but that is unrelated to this issue, so I'll close it.

Disabling contour_deformation for integrals that don't need it is some of the best speedup you can get for your money, both in compilation time and in integration time. If you have a mixed set of integration, I suggest you should automatize it somehow. Maybe if there was a reliable way that would work in the general case, we'd automatize it inside pySecDec too, but for now it's up to you.

vsht commented

Disabling contour_deformation for integrals that don't need it is some of the best speedup you can get for your money, both in compilation time and in integration time. If you have a mixed set of integration, I suggest you should automatize it somehow. Maybe if there was a reliable way that would work in the general case, we'd automatize it inside pySecDec too, but for now it's up to you.

Ok, understood!

vsht commented

Just in case you need more test cases for this behavior:

li = psd.loop_integral.LoopIntegralFromPropagators(
['p1**2', '2*p3*q', '-mg**2 + (p1 - p3)**2'],
loop_momenta = ['p1', 'p3'],
powerlist = [1, 1, 1],
dimensionality = '4 - 2*eps',
Feynman_parameters = 'x',
replacement_rules = [('q**2','mb**2')]
)

loop_package(
name = 'loopint',
loop_integral = li,
requested_orders = [2],
real_parameters = ['mb', 'mg'],
contour_deformation = True,
additional_prefactor = '(-1)*exp(2*EulerGamma*eps)',
decomposition_method = 'geometric'
)

It appears that purely real integrals involving eikonal propagators do not play nicely with contour_deformation = True.
Setting it to False makes the issue go away, as expected.

vsht commented

Still an issue with pySecDec 1.6