INCL++ ungracefully exits after a floating point error using gevgen

Question

INCL++ ungracefully exits after a floating point error using gevgen

Opened this issue a year ago · 8 comments

Whenever I run INCL++, the failure rate is high above a certain energy and the error message will not generate the event so I cannot use gevdump to investigate. I am using a slightly modified AR23_20i tune with INCL++, which I name AR23_20k. You can recreate this error with G18_11c.

Here are the setup commands I use:

source /grid/fermiapp/products/dune/setup_dune.sh
setup dk2nugenie v01_10_01k -q "e20:prof"
unsetup genie
setup genie v3_04_00d -q "e20:+inclxx:prof"

Here is the gevgen commands:
gevgen -o testk1 -n 1000 -t 1000180400 --event-generator-list Default --tune AR23_20k_00_000 --cross-sections /pnfs/dune/persistent/users/rdiurba/gxspl-NUsmall_AR23_20k_00_000.xml -e 1 -p -14 --seed 1

And this is the error:

1692714574 NOTICE Nieves : [n] <NievesQELCCPXSec.cxx::LoadConfig (428)> : RPA=1, useCoulomb=1
1692714574 INFO TransverseEnhancementFFModel : [n] <ConfigIsotopeMapUtils.cxx::LoadAllNucARangesForKey (61)> : For 0 < A < 12 -> using MagFF_RT_A = 0
1692714574 INFO TransverseEnhancementFFModel : [n] <ConfigIsotopeMapUtils.cxx::LoadAllNucARangesForKey (61)> : For 11 < A < 420 -> using MagFF_RT_A = 5.19
1692714575 INFO TransverseEnhancementFFModel : [n] <ConfigIsotopeMapUtils.cxx::LoadAllNucARangesForKey (61)> : For 0 < A < 12 -> using MagFF_RT_B = 1
1692714575 INFO TransverseEnhancementFFModel : [n] <ConfigIsotopeMapUtils.cxx::LoadAllNucARangesForKey (61)> : For 11 < A < 420 -> using MagFF_RT_B = 0.376
1692714575 WARN IntLst : [n] <MECInteractionListGenerator.cxx::CreateInteractionList (55)> : fIsCC(val) = 1
1692714575 WARN IntLst : [n] <MECInteractionListGenerator.cxx::CreateInteractionList (99)> : fIsNC = 1
1692714575 WARN IntLst : [n] <MECInteractionListGenerator.cxx::CreateInteractionList (99)> : fIsNC = 1
1692714575 WARN IntLst : [n] <MECInteractionListGenerator.cxx::CreateInteractionList (99)> : fIsNC = 1
1692714575 NOTICE Ntp : [n] <NtpWriter.cxx::NtpWriter (45)> : Run number: 0
1692714575 NOTICE Ntp : [n] <NtpWriter.cxx::NtpWriter (46)> : Requested G/ROOT tree format: [NtpMCEventRecord]
1692714575 NOTICE Ntp : [n] <NtpMCJobConfig.cxx::Load (47)> : Converting configuration registries to TFolders
1692714575 NOTICE Ntp : [n] <NtpMCJobEnv.cxx::TakeSnapshot (46)> : Taking environment snapshot and saving it in a TFolder
1692714575 NOTICE HINCLCascadeIntranuke : [n] <HINCLCascadeIntranuke.cxx::ProcessEventRecord (459)> : ************ Running HINCLCascadeIntranuke MODE INTRANUKE ************
1692714575 NOTICE HINCLCascadeIntranuke : [n] <HINCLCascadeIntranuke.cxx::TransportHadrons (500)> : Propagating hadrons within nucleus found in position = 3
1692714575 NOTICE HINCLCascadeIntranuke : [n] <HINCLCascadeIntranuke.cxx::TransportHadrons (516)> : Nucleus (A,Z) = (39, 18)
1692714575 NOTICE HINCLCascadeIntranuke : [n] <HINCLCascadeIntranuke.cxx::ProcessEventRecord (459)> : ************ Running HINCLCascadeIntranuke MODE INTRANUKE ************
1692714575 NOTICE HINCLCascadeIntranuke : [n] <HINCLCascadeIntranuke.cxx::TransportHadrons (500)> : Propagating hadrons within nucleus found in position = 3
1692714575 NOTICE HINCLCascadeIntranuke : [n] <HINCLCascadeIntranuke.cxx::TransportHadrons (516)> : Nucleus (A,Z) = (39, 18)
Collection name='genie::GHepParticles', class='genie::EventRecord', size=1000
 
 |     nu_mu_bar |            -14 |      0 |  -1 |  -1 |  -1 |  -1 |  0.000 |  0.000 |  1.000 |  1.000 |  0.000 |  
ERROR - Floating point invalid operation.
Floating point exception

Answer 1 · 2023-09-05T15:17:09.000Z

thanks for posting this. I think I see similar problem, was able to narrow it down to QE and MEC. It's not a problem with RES or DIS. Can you narrow problem down to a specific process? Error messaging isn't helpful here.

Answer 2 · 2023-09-05T15:54:16.000Z

I used the same command as above but changed --event-generator-list to CCMEC. I was able to recreate the error with the same seed. Here is the printout on Messenger_rambling.xml

1693929040 WARN IntLst : [n] <MECInteractionListGenerator.cxx::CreateInteractionList (55)> : fIsCC(val) = 1
1693929040 NOTICE Ntp : [n] <NtpWriter.cxx::NtpWriter (45)> : Run number: 0
1693929040 NOTICE Ntp : [n] <NtpWriter.cxx::NtpWriter (46)> : Requested G/ROOT tree format: [NtpMCEventRecord]
1693929040 NOTICE Ntp : [n] <NtpMCJobConfig.cxx::Load (47)> : Converting configuration registries to TFolders
1693929040 NOTICE Ntp : [n] <NtpMCJobEnv.cxx::TakeSnapshot (46)> : Taking environment snapshot and saving it in a TFolder
1693929040 NOTICE gevgen : [n] <gEvGen.cxx::GenerateEventsAtFixedInitState (312)> : 
 ** Will generate 1000 events for 
[-] [Init-State] 
 |--> probe        : PDG-code = -14 (nu_mu_bar)
 |--> nucl. target : Z = 18, A = 40, PDG-Code = 1000180400 (Ar40)
 |--> hit nucleon  : no set
 |--> hit quark    : no set
 |--> probe 4P     : (E =            0, Px =            0, Py =            0, Pz =            0)
 |--> target 4P    : (E =      37.2155, Px =            0, Py =            0, Pz =            0)
 at Ev = 1 GeV
1693929040 NOTICE gevgen : [n] <gEvGen.cxx::GenerateEventsAtFixedInitState (319)> :  *** Generating event............ 0
1693929040 NOTICE IntSel : [n] <PhysInteractionSelector.cxx::SelectInteraction (86)> : 
======================================================================================================
  Selecting an interaction for the given initial state = nu-pdg:-14;tgt-pdg:1000180400; at E = 1 GeV  
======================================================================================================
1693929040 NOTICE IntSel : [n] <PhysInteractionSelector.cxx::SelectInteraction (88)> : Computing xsecs for all relevant modeled interactions:
1693929040 NOTICE IntSel : [n] <PhysInteractionSelector.cxx::SelectInteraction (154)> : 
 |---------------------------------------------------------------------------------------------------------------|
 |                                                                      interaction | cross-section (1E-38*cm^2) |
 |---------------------------------------------------------------------------------------------------------------|
 |                                         nu:-14;tgt:1000180400;proc:Weak[CC],MEC; |                    1.66648 | 
 |---------------------------------------------------------------------------------------------------------------|

1693929040 NOTICE IntSel : [n] <PhysInteractionSelector.cxx::SelectInteraction (190)> : Selected interaction: nu:-14;tgt:1000180400;proc:Weak[CC],MEC;
1693929040 NOTICE GEVGDriver : [n] <GEVGDriver.cxx::GenerateEvent (282)> : 
==============================================================================================================
  Requesting from event generation thread: genie::EventGenerator/MEC-CC to generate the selected interaction  
==============================================================================================================

1693929040 NOTICE EventGenerator : [n] <EventGenerator.cxx::ProcessEventRecord (66)> : Generating Event...
1693929040 NOTICE EventGenerator : [n] <EventGenerator.cxx::ProcessEventRecord (69)> : Clearing the GHepVirtualListFolder
1693929040 NOTICE EventGenerator : [n] <EventGenerator.cxx::ProcessEventRecord (97)> : 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Event generation thread: genie::EventGenerator/MEC-CC -> Running module: genie::InitialStateAppender/Default  
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1693929040 INFO ISApp : [n] <InitialStateAppender.cxx::ProcessEventRecord (48)> : Adding the initial state to the event record
1693929040 INFO ISApp : [n] <InitialStateAppender.cxx::AddNeutrino (76)> : Adding neutrino [pdgc = -14]
1693929040 INFO ISApp : [n] <InitialStateAppender.cxx::AddNucleus (100)> : Adding nucleus [A = 40, Z = 18, pdg = 1000180400]
1693929040 INFO ISApp : [n] <InitialStateAppender.cxx::ProcessEventRecord (63)> : 

|------------------------------------------------------------------------------------------------------------------|
|GENIE GHEP Event Record [print level:   3]                                                                        |
|------------------------------------------------------------------------------------------------------------------|
| Idx |          Name | Ist |        PDG |   Mother  | Daughter  |      Px |      Py |      Pz |       E |      m  | 
|------------------------------------------------------------------------------------------------------------------|
|   0 |     nu_mu_bar |   0 |        -14 |  -1 |  -1 |  -1 |  -1 |   0.000 |   0.000 |   1.000 |   1.000 |   0.000 | 
|   1 |          Ar40 |   0 | 1000180400 |  -1 |  -1 |  -1 |  -1 |   0.000 |   0.000 |   0.000 |  37.216 |  37.216 | 
|------------------------------------------------------------------------------------------------------------------|
|       Fin-Init:                                                |   0.000 |   0.000 |  -1.000 | -38.216 |         | 
|------------------------------------------------------------------------------------------------------------------|
|       Vertex:      nu_mu_bar @ (x =     0.00000 m, y =     0.00000 m, z =     0.00000 m, t =    0.000000e+00 s)  |
|------------------------------------------------------------------------------------------------------------------|
| Err flag [bits:15->0] : 0000000000000000    |  1st set:                                                     none | 
| Err mask [bits:15->0] : 1111111111111111    |  Is unphysical:    NO |   Accepted:   YES                          |
|------------------------------------------------------------------------------------------------------------------|
| sig(Ev) =       1.66648e-38 cm^2  | dsig(Ev;{K_s})/dK   =     0.00000e+00 cm^2/{K}   | Weight =          1.00000 |
|------------------------------------------------------------------------------------------------------------------|

--------------------------------------------------------------------------------------------------------------
GENIE Interaction Summary
--------------------------------------------------------------------------------------------------------------
[-] [Init-State] 
 |--> probe        : PDG-code = -14 (nu_mu_bar)
 |--> nucl. target : Z = 18, A = 40, PDG-Code = 1000180400 (Ar40)
 |--> hit nucleon  : no set
 |--> hit quark    : no set
 |--> probe 4P     : (E =     1.000000, Px =     0.000000, Py =     0.000000, Pz =     1.000000)
 |--> target 4P    : (E =    37.215526, Px =     0.000000, Py =     0.000000, Pz =     0.000000)

[-] [Process-Info]  
 |--> Interaction : Weak[CC]
 |--> Scattering  : MEC
[-] [Kinematics]
[-] [Exclusive Process Info] 
 |--> charm prod.  : false |--> strange prod.  : false
 |--> f/s nucleons : N(p) = 0 N(n) = 0
 |--> f/s pions    : N(pi^0) = 0 N(pi^+) = 0 N(pi^-) = 0
 |--> f/s Other    : N(gamma) = 0 N(Rho^0) = 0 N(Rho^+) = 0 N(Rho^-) = 0
 |--> resonance    : [not set]
 |--> final quark prod.  : false
 |--> final lepton prod.  : false
--------------------------------------------------------------------------------------------------------------


1693929040 NOTICE EventGenerator : [n] <EventGenerator.cxx::ProcessEventRecord (97)> : 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Event generation thread: genie::EventGenerator/MEC-CC -> Running module: genie::VertexGenerator/Default  
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1693929040 INFO Vtx : [n] <VertexGenerator.cxx::GenerateVertex (146)> : Generating vertex according to a realistic nuclear density profile
1693929040 INFO Vtx : [n] <VertexGenerator.cxx::GenerateVertex (211)> : Generated vtx @ r = 1.84035 fm / (x = 1.30244, y = -0.00785201, z = -1.30019)
1693929040 INFO Vtx : [n] <VertexGenerator.cxx::ProcessEventRecord (73)> : Setting vertex position for: nu_mu_bar
1693929040 NOTICE EventGenerator : [n] <EventGenerator.cxx::ProcessEventRecord (97)> : 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Event generation thread: genie::EventGenerator/MEC-CC -> Running module: genie::MECGenerator/Default  
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Collection name='genie::GHepParticles', class='genie::EventRecord', size=1000
 
 |     nu_mu_bar |            -14 |      0 |  -1 |  -1 |  -1 |  -1 |  0.000 |  0.000 |  1.000 |  1.000 |  0.000 |  
ERROR - Floating point invalid operation.
Floating point exception

Answer 3 · 2023-09-05T16:03:11.000Z

yes, that is the idea. Use CCQE, CCMEC, CCRES, and CCDIS separately to show which simulations are affected. By the way, I use G18_10c events on my laptop so it's likely more general. To be specific, problem seems to be with 1st event of a specific process.

Answer 4 · 2023-09-06T08:50:12.000Z

I am sorry but I do not know if I have access to a G18_10a xml spline, hence I am using AR23_20i splines. I have tried all CC topologies and they all work except for CCMEC.

Answer 5 · 2023-11-09T13:54:45.000Z

The fix should be in #360

Answer 6 · 2023-11-09T14:05:21.000Z

Hey Marco, Thank you for the update. I have a few questions and then I will try it out in a bit. Are there any additional fixes planned or does this extend from the MEC issue to the O11 issue Julia and I saw? Is a tagged release planned in the near future? Thank you to the whole GENIE team for all the hard work! Thanks, Richie

…

________________________________ From: Marco Roda ***@***.***> Sent: Thursday, November 9, 2023 2:54 PM To: GENIE-MC/Generator ***@***.***> Cc: rdiurba ***@***.***>; Author ***@***.***> Subject: Re: [GENIE-MC/Generator] INCL++ ungracefully exits after a floating point error using gevgen (Issue #320) The fix should be in #360<#360> — Reply to this email directly, view it on GitHub<#320 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOZJTLFBY3TSOGYAPGUEX5LYDTODBAVCNFSM6AAAAAA32BAJHSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBTHA3TMNJRGI>. You are receiving this because you authored the thread.Message ID: ***@***.***>

Answer 7 · 2023-11-09T14:11:30.000Z

This should fix everything. Although we have yet to test it.

We think we will release a new tag before the Christmas holidays. We know for sure that the next release will require this fix anyway.

Answer 8 · 2023-11-13T09:19:35.000Z

Hey Marco and GENIE, I ran some CCMEC events at 5 GeV for AR23 with the three different FSIs and I was able to run CCMEC on INCL++, which is a good sign. Furthermore, the double-banding for Geant4 that Callum saw seems to be fixed in the images attached. I was able to run gevgen_had on carbon, iron, and lead on INCL++ and Geant4. I did not see any bugs, whereas before carbon and lead would fail due to various isotopes being generated. I will keep investigating but so far progress has been made. Thanks, Richie

…

________________________________ From: Marco Roda ***@***.***> Sent: Thursday, November 9, 2023 3:11 PM To: GENIE-MC/Generator ***@***.***> Cc: rdiurba ***@***.***>; Author ***@***.***> Subject: Re: [GENIE-MC/Generator] INCL++ ungracefully exits after a floating point error using gevgen (Issue #320) This should fix everything. Although we have yet to test it. We think we will release a new tag before the Christmas holidays. We know for sure that the next release will require this fix anyway. — Reply to this email directly, view it on GitHub<#320 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOZJTLHTVWOBQ64KLZTTE3TYDTQB3AVCNFSM6AAAAAA32BAJHSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBTHEYDONBRGM>. You are receiving this because you authored the thread.