BUG: testing framework somehow breaks dump_io
Closed this issue · 3 comments
In GitLab by @neutrinoceros on May 2, 2023, 16:47
When trying to run testme.py script locally I get Python to crash when reading a dump file.
Here a full traceback
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/robcleme/dev/idefix-project/idefix/test/Planet/PlanetMigration2D/testme.py:39 in <module> │
│ │
│ 36 if(test.check): │
│ 37 │ test.checkOnly(filename=name,tolerance=tolerance) │
│ 38 else: │
│ ❱ 39 │ testMe(test) │
│ 40 else: │
│ 41 test.noplot = True │
│ 42 test.vectPot=False │
│ │
│ /Users/robcleme/dev/idefix-project/idefix/test/Planet/PlanetMigration2D/testme.py:28 in testMe │
│ │
│ 25 │ if test.init and not test.mpi: │
│ 26 │ test.makeReference(filename=name) │
│ 27 │ test.standardTest() │
│ ❱ 28 │ test.nonRegressionTest(filename=name,tolerance=tolerance) │
│ 29 │
│ 30 │
│ 31 test=tst.idfxTest() │
│ │
│ /Users/robcleme/dev/idefix-project/idefix/pytools/idfx_test.py:305 in nonRegressionTest │
│ │
│ 302 │ if not(os.path.exists(filetest)): │
│ 303 │ raise Exception("Test file "+fileref+ " doesn't exist") │
│ 304 │ │
│ ❱ 305 │ Vref=readDump(fileref) │
│ 306 │ Vtest=readDump(filetest) │
│ 307 │ error=self._computeError(Vref,Vtest) │
│ 308 │ if error > tolerance: │
│ │
│ /Users/robcleme/dev/idefix-project/idefix/pytools/dump_io.py:124 in readDump │
│ │
│ 121 │
│ 122 # public API │
│ 123 def readDump(filename): │
│ ❱ 124 │ return DumpDataset(filename) │
│ 125 │
│ │
│ /Users/robcleme/dev/idefix-project/idefix/pytools/dump_io.py:74 in __init__ │
│ │
│ 71 │ │ self.filename = os.path.abspath(filename) │
│ 72 │ │ self.metadata = {} │
│ 73 │ │ with open(filename, "rb") as fh: │
│ ❱ 74 │ │ │ self._read_header(fh) │
│ 75 │ │ │ self._read_fields(fh) │
│ 76 │ │
│ 77 │ def _read_header(self, fh): │
│ │
│ /Users/robcleme/dev/idefix-project/idefix/pytools/dump_io.py:79 in _read_header │
│ │
│ 76 │ │
│ 77 │ def _read_header(self, fh): │
│ 78 │ │ q = fh.read(HEADER_SIZE) │
│ ❱ 79 │ │ n = q.index(b"\x00") │
│ 80 │ │ self.header = q[:n].decode("utf-8") │
│ 81 │ │ │
│ 82 │ │ match = re.match(header_regexp, self.header) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: subsection not found
> /Users/robcleme/dev/idefix-project/idefix/pytools/dump_io.py(79)_read_header()
77 def _read_header(self, fh):
78 q = fh.read(HEADER_SIZE)
---> 79 n = q.index(b"\x00")
80 self.header = q[:n].decode("utf-8")
81
ipdb> print(q)
b'version https://git-lfs.github.com/spec/v1\noid sha256:1346a943022e012a1ba6a6169583b7a4dd105130117bb0f8ee1f16dbf0bfda67\nsize 4742'
What seems to be happening is that the header overflows so it ends up not being null-terminated. I currently have no idea why this happens only within the context of testme.py scripts, but so far the crash is systematic on my machine.
In GitLab by @neutrinoceros on May 2, 2023, 17:52
So it seems that the problematic header is in fact found in the reference file that is tracked with git-lfs, and its length is >130 char, when HEADERSIZE=128.
What I don't get yet is why this problem is only showing on my machine and not in CI
In GitLab by @neutrinoceros on May 5, 2023, 23:42
mentioned in commit bd72dca
In GitLab by @neutrinoceros on Jun 17, 2023, 17:41
turns out my git-lfs install was broken. The current version of the reference file doesn't have this problem.