root-project/root

STL collection object in TFile not compatible across OSX and Linux

Opened this issue · 7 comments

ktf commented

Check duplicate issues.

  • Checked for duplicates

Description

As reported this summer in https://root-forum.cern.ch/t/stl-collection-object-in-tfile-not-compatible-across-osx-and-linux/60321

Using:

#include <array>
#include <iostream>
#include <vector>

#include "TFile.h"

#pragma link C++ class std::vector<std::array<ULong64_t, 2>>+ ;

void writeVector() {
  std::vector < std::array < ULong64_t, 2>> test { { 1ull, 2ull }, { 3ull, 4ull }, { 5ull, 6ull } };
  TFile f("test.root", "RECREATE");
  f.WriteObject(&test, "test");
  f.Close();
}

void readVector() {
  TFile f("test.root");
  auto test = f.Get<std::vector<std::array < ULong64_t, 2> > >("test");

  for (auto &a: *test) {
    std::cout << a[0] << " " << a[1] << std::endl;
  }
  f.Close();
}

to write on one platform and then read the result back on the other results in:

Error in <TBufferFile::ReadVersion>: Could not find the StreamerInfo with a checksum of 0x44178e08 for the class "array<ULong64_t,2>" in test.root.
Error in <TBufferFile::CheckByteCount>: object of class array<ULong64_t,2> read too few bytes: 6 instead of 22
Error in <TBufferFile::ReadVersion>: Could not find the StreamerInfo with a checksum of 0x44178e08 for the class "array<ULong64_t,2>" in test.root.
Error in <TBufferFile::CheckByteCount>: object of class array<ULong64_t,2> read too few bytes: 6 instead of 22
Error in <TBufferFile::ReadVersion>: Could not find the StreamerInfo with a checksum of 0x44178e08 for the class "array<ULong64_t,2>" in test.root.
Error in <TBufferFile::CheckByteCount>: object of class array<ULong64_t,2> read too few bytes: 6 instead of 22

Similarly, if I adapt the above to be an std::unordered_map rather than a vector of pairs it crashes with:

[/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info)
[/Users/ktf/src/sw/BUILD/44309b19dcd447bff3ead99ccd49cb399f7edcaa/O2/compiled_macros/Users/ktf/src/sw/BUILD/44309b19dcd447bff3ead99ccd49cb399f7edcaa/O2/foo_C.so] readVector() (no
debug info)
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCling.6.32.06.so] cling::IncrementalExecutor::executeWrapper(llvm::StringRef, cling::Value*) const (no debug info)
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCling.6.32.06.so] cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) (no debug info)
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCling.6.32.06.so] cling::Interpreter::EvaluateInternal(std::__1::basic_string<char, std::__1::char_traits<char>, std::__
1::allocator<char>> const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) (no debug info)
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCling.6.32.06.so] cling::Interpreter::process(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::alloca
tor<char>> const&, cling::Value*, cling::Transaction**, bool) (no debug info)
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCling.6.32.06.so] cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*,
bool) (no debug info)
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCling.6.32.06.so] HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&,
 cling::Value*) /Users/ktf/src/sw/SOURCES/ROOT/10b8d555b9/0/core/metacling/src/TCling.cxx:2447
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCling.6.32.06.so] TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) /Users/ktf/src/sw/SOURCES/ROOT/10b8d555b9/
0/core/metacling/src/TCling.cxx:0
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libRint.6.32.06.so] TRint::ProcessLineNr(char const*, char const*, int*) /Users/ktf/src/sw/SOURCES/ROOT/10b8d555b9/0/core/r
int/src/TRint.cxx:0
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libRint.6.32.06.so] TRint::HandleTermInput() /Users/ktf/src/sw/SOURCES/ROOT/10b8d555b9/0/core/rint/src/TRint.cxx:649
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCore.6.32.06.so] TUnixSystem::CheckDescriptors() /Users/ktf/src/sw/SOURCES/ROOT/10b8d555b9/0/core/unix/src/TUnixSystem.c
xx:0
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCore.6.32.06.so] TMacOSXSystem::DispatchOneEvent(bool) /Users/ktf/src/sw/SOURCES/ROOT/10b8d555b9/0/core/macosx/src/TMacO
SXSystem.mm:378
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCore.6.32.06.so] TSystem::InnerLoop() /Users/ktf/src/sw/SOURCES/ROOT/10b8d555b9/0/core/base/src/TSystem.cxx:404
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCore.6.32.06.so] TSystem::Run() /Users/ktf/src/sw/SOURCES/ROOT/10b8d555b9/0/core/base/src/TSystem.cxx:354
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libCore.6.32.06.so] TApplication::Run(bool) /Users/ktf/src/sw/SOURCES/ROOT/10b8d555b9/0/core/base/src/TApplication.cxx:1887
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/lib/libRint.6.32.06.so] TRint::Run(bool) /Users/ktf/src/sw/SOURCES/ROOT/10b8d555b9/0/core/rint/src/TRint.cxx:0
[/Users/ktf/src/sw/osx_arm64/ROOT/10b8d555b9-local1/bin/root.exe] main /Users/ktf/src/sw/SOURCES/ROOT/10b8d555b9/0/main/src/rmain.cxx:86

Running on the same platform works as expected.

Reproducer

See description

ROOT version

6.32.6 built with alibuild

Installation method

alibuild

Operating system

macOS + linux (lxplus)

Additional context

This is preventing us to read some calibration objects which were created during datataking and use them to perform analysis tasks on macOS.

pcanal commented

Does it happens to anything beside a STL collection of std::array? This combination is not yet supported (and sadly not yet explicitly rejected either :( ). [ the recommendation it to use std::collection< struct_wrapping_std_array<..>> ]

ktf commented

Yes, with unordered_map<uin64_t, double > , which is how we (re)discovered this today.

pcanal commented

I would not expect the same error message (type) with that. It is likely that uin64_t has a different underlying type on Linux and Macos and hence result in effectively different class name for unordered_map<uin64_t, double >.
What is the actual error message in this case?

ktf commented 
root [0]
Attaching file 6382c81b-b32b-11ef-81d0-0aa202c71b9a as _file0...
Warning in <TStreamerInfo::BuildCheck>:
   The StreamerInfo of class o2::grp::GRPEnvVariables read from file 6382c81b-b32b-11ef-81d0-0aa202c71b9a
   has the same version (=1) as the active class but a different checksum.
   You should update the version to ClassDef(o2::grp::GRPEnvVariables,2).
   Do not try to write objects with the current class definition,
   the files will not be readable.

Warning in <TStreamerInfo::CompareContent>: The following data member of
the on-file layout version 1 of class 'o2::grp::GRPEnvVariables' differs from
the in-memory layout version 1:
   unordered_map<string,vector<pair<unsigned long,double> > > mEnvVars; //
vs
   unordered_map<string,vector<pair<ULong64_t,double> > > mEnvVars; //
(TFile *) 0x12db0c0d0
root [1] .q

is the original error in the production code.

pcanal commented

Alright. That error is what I expect. If you are using std::uin64_t with the current ROOT code, the schema will be different on different platforms. Until we had support for this typedefs (in the plan for work for 2025), your best solution is to use the ROOT typedef instead (i.e. ULong64_t) or (for this case) unsigned long long would also work. [Two other solutions might be to remove the class version from that class and just rely on checksum or to make the class version platform dependent]

ktf commented

If we substitute with unsigned long long, would that require bumping the version to read it correctly?

pcanal commented

Yes, the only way to avoid the warnings in most cases is to change the type and increase the version number. (You might still get the warning when trying to read in the same process files written on both mac and linux - because 2 conflicting version 1 would be read)