Memory issue when writing long (long) comment for HIERARCH keyword

Question

Memory issue when writing long (long) comment for HIERARCH keyword

kabasset opened this issue 3 years ago · 7 comments

Internally reported by @grizonnetm

I found some strange memory issue (invalid write) when I create a record with a hierarchical keyword (more than 8 characters) and a long (very long) comment.
I don't know exactly if this is a limitation from fits or CfitsIO and I had some trouble to find a way to reproduce the issue as I did not have the same output if I run all the tests, unit test per unit test, with valgrind...
I've created a branch in el_fitsIO which update unit tests which use the checkHierarchKeywordIsReadBack function and increase the size of the comment associated to the hierarchical keyword:
https://gitlab.euclid-sgs.uk/EuclidLibs/EL_FitsIO/-/commit/25e82146e8bc35cb3608105d766965a6078bd8e2
Branch:
https://gitlab.euclid-sgs.uk/EuclidLibs/EL_FitsIO/-/tree/bug-long-comment-with-hierarch-keyword
In this branch, I've let only 1 test related to hierarchical keyword to illustrate
And I've attached the valgrind log (valgrind ./build.x86_64-conda_cos6-gcc73-o2g/bin/EL_FitsFile_RecordHdu_test).
I'll update the issue if I get more information.

Answer 1 · 2022-01-22T20:09:33.000Z

Log:

==99493== Memcheck, a memory error detector
==99493== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==99493== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==99493== Command: ./build.x86_64-conda_cos6-gcc73-o2g/bin/EL_FitsFile_RecordHdu_test
==99493== 
==99493== Invalid write of size 4
==99493==    at 0x4FF809D: __cxa_init_primary_exception (eh_throw.cc:63)
==99493==    by 0x4FF810C: __cxa_throw (eh_throw.cc:84)
==99493==    by 0x409CA8A: Euclid::Cfitsio::CfitsioError::mayThrow(int, fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (ErrorWrapper.cpp:77)
==99493==    by 0x409F853: Euclid::Cfitsio::Header::recordTypeid(fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (HeaderWrapper.cpp:375)
==99493==    by 0x40A0772: Euclid::FitsIO::Record<boost::any> Euclid::Cfitsio::Header::parseRecord<boost::any>(fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (HeaderWrapper.cpp:161)
==99493==    by 0x12C4C2: parse<boost::any> (Header.hpp:31)
==99493==    by 0x12C4C2: parseRecord<boost::any> (RecordHdu.hpp:36)
==99493==    by 0x12C4C2: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test::test_method() (RecordHdu_test.cpp:175)
==99493==    by 0x130498: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test_invoker() (RecordHdu_test.cpp:168)
==99493==    by 0x4E5C22D: boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A31C: boost::execution_monitor::catch_signals(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A403: boost::execution_monitor::execute(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5ABCD: boost::execution_monitor::vexecute(boost::function<void ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E8C6C1: boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned int) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==  Address 0x891de80 is 1 bytes after a block of size 31 free'd
==99493==    at 0x402EF5B: operator delete(void*) (vg_replace_malloc.c:586)
==99493==    by 0x148490: deallocate (new_allocator.h:125)
==99493==    by 0x148490: deallocate (alloc_traits.h:462)
==99493==    by 0x148490: _M_destroy (basic_string.h:226)
==99493==    by 0x148490: _M_dispose (basic_string.h:221)
==99493==    by 0x148490: ~basic_string (basic_string.h:647)
==99493==    by 0x148490: void Euclid::Cfitsio::Header::writeRecord<int>(fitsfile*, Euclid::FitsIO::Record<int> const&) (HeaderWrapper.hpp:184)
==99493==    by 0x40A218D: writeRecordAnyImpl<int> (HeaderWrapper.cpp:202)
==99493==    by 0x40A218D: void Euclid::Cfitsio::Header::writeRecord<boost::any>(fitsfile*, Euclid::FitsIO::Record<boost::any> const&) (HeaderWrapper.cpp:213)
==99493==    by 0x12C3BB: write<boost::any> (Header.hpp:118)
==99493==    by 0x12C3BB: operator()<Euclid::FitsIO::Record<boost::any> > (Header.hpp:178)
==99493==    by 0x12C3BB: seqForeach<Euclid::FitsIO::Record<boost::any>, Euclid::FitsIO::Header::writeSeqIn(const std::vector<std::__cxx11::basic_string<char> >&, TSeq&&) const [with Euclid::FitsIO::RecordMode Mode = (Euclid::FitsIO::RecordMode)2; TSeq = const std::vector<Euclid::FitsIO::Record<boost::any> >&]::<lambda(const auto:3&)>&> (DataUtils.h:185)
==99493==    by 0x12C3BB: writeSeqIn<(Euclid::FitsIO::RecordMode)2, const std::vector<Euclid::FitsIO::Record<boost::any> >&> (Header.hpp:181)
==99493==    by 0x12C3BB: writeRecords<boost::any> (RecordHdu.hpp:108)
==99493==    by 0x12C3BB: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test::test_method() (RecordHdu_test.cpp:174)
==99493==    by 0x130498: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test_invoker() (RecordHdu_test.cpp:168)
==99493==    by 0x4E5C22D: boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A31C: boost::execution_monitor::catch_signals(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A403: boost::execution_monitor::execute(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5ABCD: boost::execution_monitor::vexecute(boost::function<void ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E8C6C1: boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned int) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E6A885: boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned int, boost::unit_test::framework::state::random_generator_helper const*) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E6AA05: boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned int, boost::unit_test::framework::state::random_generator_helper const*) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==  Block was alloc'd at
==99493==    at 0x402DE3F: operator new(unsigned long) (vg_replace_malloc.c:344)
==99493==    by 0x4119A81: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_program_options.so.1.69.0)
==99493==    by 0x141DAE: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::operator+<char, std::char_traits<char>, std::allocator<char> >(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (basic_string.tcc:1165)
==99493==    by 0x14846E: void Euclid::Cfitsio::Header::writeRecord<int>(fitsfile*, Euclid::FitsIO::Record<int> const&) (HeaderWrapper.hpp:184)
==99493==    by 0x40A218D: writeRecordAnyImpl<int> (HeaderWrapper.cpp:202)
==99493==    by 0x40A218D: void Euclid::Cfitsio::Header::writeRecord<boost::any>(fitsfile*, Euclid::FitsIO::Record<boost::any> const&) (HeaderWrapper.cpp:213)
==99493==    by 0x12C3BB: write<boost::any> (Header.hpp:118)
==99493==    by 0x12C3BB: operator()<Euclid::FitsIO::Record<boost::any> > (Header.hpp:178)
==99493==    by 0x12C3BB: seqForeach<Euclid::FitsIO::Record<boost::any>, Euclid::FitsIO::Header::writeSeqIn(const std::vector<std::__cxx11::basic_string<char> >&, TSeq&&) const [with Euclid::FitsIO::RecordMode Mode = (Euclid::FitsIO::RecordMode)2; TSeq = const std::vector<Euclid::FitsIO::Record<boost::any> >&]::<lambda(const auto:3&)>&> (DataUtils.h:185)
==99493==    by 0x12C3BB: writeSeqIn<(Euclid::FitsIO::RecordMode)2, const std::vector<Euclid::FitsIO::Record<boost::any> >&> (Header.hpp:181)
==99493==    by 0x12C3BB: writeRecords<boost::any> (RecordHdu.hpp:108)
==99493==    by 0x12C3BB: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test::test_method() (RecordHdu_test.cpp:174)
==99493==    by 0x130498: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test_invoker() (RecordHdu_test.cpp:168)
==99493==    by 0x4E5C22D: boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A31C: boost::execution_monitor::catch_signals(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A403: boost::execution_monitor::execute(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5ABCD: boost::execution_monitor::vexecute(boost::function<void ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E8C6C1: boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned int) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493== 
==99493== Invalid write of size 8
==99493==    at 0x4FF80A4: __cxa_init_primary_exception (eh_throw.cc:64)
==99493==    by 0x4FF810C: __cxa_throw (eh_throw.cc:84)
==99493==    by 0x409CA8A: Euclid::Cfitsio::CfitsioError::mayThrow(int, fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (ErrorWrapper.cpp:77)
==99493==    by 0x409F853: Euclid::Cfitsio::Header::recordTypeid(fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (HeaderWrapper.cpp:375)
==99493==    by 0x40A0772: Euclid::FitsIO::Record<boost::any> Euclid::Cfitsio::Header::parseRecord<boost::any>(fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (HeaderWrapper.cpp:161)
==99493==    by 0x12C4C2: parse<boost::any> (Header.hpp:31)
==99493==    by 0x12C4C2: parseRecord<boost::any> (RecordHdu.hpp:36)
==99493==    by 0x12C4C2: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test::test_method() (RecordHdu_test.cpp:175)
==99493==    by 0x130498: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test_invoker() (RecordHdu_test.cpp:168)
==99493==    by 0x4E5C22D: boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A31C: boost::execution_monitor::catch_signals(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A403: boost::execution_monitor::execute(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5ABCD: boost::execution_monitor::vexecute(boost::function<void ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E8C6C1: boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned int) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==  Address 0x891de90 is 17 bytes after a block of size 31 free'd
==99493==    at 0x402EF5B: operator delete(void*) (vg_replace_malloc.c:586)
==99493==    by 0x148490: deallocate (new_allocator.h:125)
==99493==    by 0x148490: deallocate (alloc_traits.h:462)
==99493==    by 0x148490: _M_destroy (basic_string.h:226)
==99493==    by 0x148490: _M_dispose (basic_string.h:221)
==99493==    by 0x148490: ~basic_string (basic_string.h:647)
==99493==    by 0x148490: void Euclid::Cfitsio::Header::writeRecord<int>(fitsfile*, Euclid::FitsIO::Record<int> const&) (HeaderWrapper.hpp:184)
==99493==    by 0x40A218D: writeRecordAnyImpl<int> (HeaderWrapper.cpp:202)
==99493==    by 0x40A218D: void Euclid::Cfitsio::Header::writeRecord<boost::any>(fitsfile*, Euclid::FitsIO::Record<boost::any> const&) (HeaderWrapper.cpp:213)
==99493==    by 0x12C3BB: write<boost::any> (Header.hpp:118)
==99493==    by 0x12C3BB: operator()<Euclid::FitsIO::Record<boost::any> > (Header.hpp:178)
==99493==    by 0x12C3BB: seqForeach<Euclid::FitsIO::Record<boost::any>, Euclid::FitsIO::Header::writeSeqIn(const std::vector<std::__cxx11::basic_string<char> >&, TSeq&&) const [with Euclid::FitsIO::RecordMode Mode = (Euclid::FitsIO::RecordMode)2; TSeq = const std::vector<Euclid::FitsIO::Record<boost::any> >&]::<lambda(const auto:3&)>&> (DataUtils.h:185)
==99493==    by 0x12C3BB: writeSeqIn<(Euclid::FitsIO::RecordMode)2, const std::vector<Euclid::FitsIO::Record<boost::any> >&> (Header.hpp:181)
==99493==    by 0x12C3BB: writeRecords<boost::any> (RecordHdu.hpp:108)
==99493==    by 0x12C3BB: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test::test_method() (RecordHdu_test.cpp:174)
==99493==    by 0x130498: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test_invoker() (RecordHdu_test.cpp:168)
==99493==    by 0x4E5C22D: boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A31C: boost::execution_monitor::catch_signals(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A403: boost::execution_monitor::execute(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5ABCD: boost::execution_monitor::vexecute(boost::function<void ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E8C6C1: boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned int) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E6A885: boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned int, boost::unit_test::framework::state::random_generator_helper const*) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E6AA05: boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned int, boost::unit_test::framework::state::random_generator_helper const*) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==  Block was alloc'd at
==99493==    at 0x402DE3F: operator new(unsigned long) (vg_replace_malloc.c:344)
==99493==    by 0x4119A81: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_program_options.so.1.69.0)
==99493==    by 0x141DAE: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::operator+<char, std::char_traits<char>, std::allocator<char> >(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (basic_string.tcc:1165)
==99493==    by 0x14846E: void Euclid::Cfitsio::Header::writeRecord<int>(fitsfile*, Euclid::FitsIO::Record<int> const&) (HeaderWrapper.hpp:184)
==99493==    by 0x40A218D: writeRecordAnyImpl<int> (HeaderWrapper.cpp:202)
==99493==    by 0x40A218D: void Euclid::Cfitsio::Header::writeRecord<boost::any>(fitsfile*, Euclid::FitsIO::Record<boost::any> const&) (HeaderWrapper.cpp:213)
==99493==    by 0x12C3BB: write<boost::any> (Header.hpp:118)
==99493==    by 0x12C3BB: operator()<Euclid::FitsIO::Record<boost::any> > (Header.hpp:178)
==99493==    by 0x12C3BB: seqForeach<Euclid::FitsIO::Record<boost::any>, Euclid::FitsIO::Header::writeSeqIn(const std::vector<std::__cxx11::basic_string<char> >&, TSeq&&) const [with Euclid::FitsIO::RecordMode Mode = (Euclid::FitsIO::RecordMode)2; TSeq = const std::vector<Euclid::FitsIO::Record<boost::any> >&]::<lambda(const auto:3&)>&> (DataUtils.h:185)
==99493==    by 0x12C3BB: writeSeqIn<(Euclid::FitsIO::RecordMode)2, const std::vector<Euclid::FitsIO::Record<boost::any> >&> (Header.hpp:181)
==99493==    by 0x12C3BB: writeRecords<boost::any> (RecordHdu.hpp:108)
==99493==    by 0x12C3BB: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test::test_method() (RecordHdu_test.cpp:174)
==99493==    by 0x130498: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test_invoker() (RecordHdu_test.cpp:168)
==99493==    by 0x4E5C22D: boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A31C: boost::execution_monitor::catch_signals(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A403: boost::execution_monitor::execute(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5ABCD: boost::execution_monitor::vexecute(boost::function<void ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E8C6C1: boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned int) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493== 
==99493== Invalid write of size 8
==99493==    at 0x4FF80A8: __cxa_init_primary_exception (eh_throw.cc:65)
==99493==    by 0x4FF810C: __cxa_throw (eh_throw.cc:84)
==99493==    by 0x409CA8A: Euclid::Cfitsio::CfitsioError::mayThrow(int, fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (ErrorWrapper.cpp:77)
==99493==    by 0x409F853: Euclid::Cfitsio::Header::recordTypeid(fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (HeaderWrapper.cpp:375)
==99493==    by 0x40A0772: Euclid::FitsIO::Record<boost::any> Euclid::Cfitsio::Header::parseRecord<boost::any>(fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (HeaderWrapper.cpp:161)
==99493==    by 0x12C4C2: parse<boost::any> (Header.hpp:31)
==99493==    by 0x12C4C2: parseRecord<boost::any> (RecordHdu.hpp:36)
==99493==    by 0x12C4C2: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test::test_method() (RecordHdu_test.cpp:175)
==99493==    by 0x130498: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test_invoker() (RecordHdu_test.cpp:168)
==99493==    by 0x4E5C22D: boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A31C: boost::execution_monitor::catch_signals(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A403: boost::execution_monitor::execute(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5ABCD: boost::execution_monitor::vexecute(boost::function<void ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E8C6C1: boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned int) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==  Address 0x891de98 is 24 bytes after a block of size 32 in arena "client"
==99493== 

valgrind: m_mallocfree.c:307 (get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed.
valgrind: Heap block lo/hi size mismatch: lo = 96, hi = 67750560.
This is probably caused by your program erroneously writing past the
end of a heap block and corrupting heap metadata.  If you fix any
invalid writes reported by Memcheck, this assertion failure will
probably go away.  Please try that before reporting this as a bug.


host stacktrace:
==99493==    at 0x5805457B: show_sched_status_wrk (m_libcassert.c:388)
==99493==    by 0x580546A7: report_and_quit (m_libcassert.c:459)
==99493==    by 0x5805483F: vgPlain_assert_fail (m_libcassert.c:525)
==99493==    by 0x5805D652: get_bszB_as_is (m_mallocfree.c:305)
==99493==    by 0x5805D652: get_bszB (m_mallocfree.c:315)
==99493==    by 0x5805D652: get_pszB (m_mallocfree.c:389)
==99493==    by 0x5805D652: vgPlain_describe_arena_addr (m_mallocfree.c:1592)
==99493==    by 0x580493CB: vgPlain_describe_addr (m_addrinfo.c:187)
==99493==    by 0x58048207: vgMemCheck_update_Error_extra (mc_errors.c:1187)
==99493==    by 0x5804D7C7: vgPlain_maybe_record_error (m_errormgr.c:823)
==99493==    by 0x580472BD: vgMemCheck_record_address_error (mc_errors.c:767)
==99493==    by 0x1003EC5DC2: ???
==99493==    by 0x10035B2F2F: ???
==99493==    by 0x10020083AF: ???
==99493==    by 0x10035B2F17: ???
==99493==    by 0x10035B2F2F: ???
==99493==    by 0x10035B2F3F: ???
==99493==    by 0x592B8BBF: ???
==99493==    by 0x5C63: ???
==99493==    by 0x10057F009F: ???
==99493==    by 0x5C63: ???

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable (lwpid 99493)
==99493==    at 0x4FF80B1: __cxa_init_primary_exception (eh_throw.cc:66)
==99493==    by 0x4FF810C: __cxa_throw (eh_throw.cc:84)
==99493==    by 0x409CA8A: Euclid::Cfitsio::CfitsioError::mayThrow(int, fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (ErrorWrapper.cpp:77)
==99493==    by 0x409F853: Euclid::Cfitsio::Header::recordTypeid(fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (HeaderWrapper.cpp:375)
==99493==    by 0x40A0772: Euclid::FitsIO::Record<boost::any> Euclid::Cfitsio::Header::parseRecord<boost::any>(fitsfile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (HeaderWrapper.cpp:161)
==99493==    by 0x12C4C2: parse<boost::any> (Header.hpp:31)
==99493==    by 0x12C4C2: parseRecord<boost::any> (RecordHdu.hpp:36)
==99493==    by 0x12C4C2: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test::test_method() (RecordHdu_test.cpp:175)
==99493==    by 0x130498: RecordHdu_test::subset_of_vector_of_any_records_is_read_back_test_invoker() (RecordHdu_test.cpp:168)
==99493==    by 0x4E5C22D: boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A31C: boost::execution_monitor::catch_signals(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5A403: boost::execution_monitor::execute(boost::function<int ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E5ABCD: boost::execution_monitor::vexecute(boost::function<void ()> const&) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E8C6C1: boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned int) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E6A885: boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned int, boost::unit_test::framework::state::random_generator_helper const*) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E6AA05: boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned int, boost::unit_test::framework::state::random_generator_helper const*) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E6AA05: boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned int, boost::unit_test::framework::state::random_generator_helper const*) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E62F9C: boost::unit_test::framework::run(unsigned long, bool) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x4E8A657: boost::unit_test::unit_test_main(bool (*)(), int, char**) (in /cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.1/lib/libboost_unit_test_framework.so.1.69.0)
==99493==    by 0x53E4554: (below main) (in /usr/lib64/libc-2.17.so)
client stack range: [0x1FFEFFA000 0x1FFF000FFF] client SP: 0x1FFEFFCE60
valgrind stack range: [0x10034B3000 0x10035B2FFF] top usage: 11768 of 1048576

Answer 2 · 2022-01-22T20:11:32.000Z

Have to check how to get the size of the comment string, as for the value.

Answer 3 · 2022-02-03T09:11:44.000Z

No known CFITSIO routine to get the comment size (mail just sent to CFITSIO maintainers, awaiting answer). The only workaround I have in mind is getting the header size and using it to allocate the comment, but this means wasting a lot of bytes for each long string record reading.

Answer 4 · 2022-02-03T10:01:50.000Z

Unit test added

Issue found: long comments are truncated; Yet, there is no memory issue.

Answer 5 · 2022-02-03T14:25:11.000Z

OK, I read too fast: your issue is not only related to long comments, but long comments with HIERARCH keywords.
I'll have to investigate a bit...

Answer 6 · 2022-02-03T14:45:20.000Z

I've written a unit test to try to reproduce your issue, but was not able to.
Here is what I do:

create an int record with 9-char keyword and > 68-char comment
write the record
parse the record back
check that the values are equal
check that the comments are different (this is the expected behavior since int records cannot be multiline)

I see no memory issue.

@grizonnetm Do you know if you still have the issue with the refactored EleFits (major version > 3) ?
If that's the case, I should setup a more complex test which involves a RecordSeq.

Answer 7 · 2022-02-03T17:28:05.000Z

I confirm that I can't reproduce the issue with Elefits 5.0. Feel free to close it, thanks