simdb example

[danbone]

I've tried to copy some of the simdb examples but I'm not having much luck getting things to work.

Would it be possible to implement an example using olympia?

Mainly I've followed
https://github.com/sparcians/map/blob/master/helios/plato/docs/capturing_hdf5_with_sparta.md

Which I've gotten into a state where it actually produces a h5 file but I can't actually read it using pandas

(this is the example under plato/demo/data, same thing occurs in my h5 file to)

[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.read_hdf('test.hdf5')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/vl/edatools/intern/python/3.8.3/CentOS-7/lib/python3.8/site-packages/pandas/io/pytables.py", line 398, in read_hdf
    raise ValueError(
ValueError: Dataset(s) incompatible with Pandas data types, not table, or no datasets found in HDF5 file.```

[klingaard]

@aarongchan I know you got simdb up and running with SQLite in the past, did you play around with the HDF5 support at all?

[aarongchan]

@aarongchan I know you got simdb up and running with SQLite in the past, did you play around with the HDF5 support at all?

I never tried using HDF5. One thing I thought of was there might be compression being used in HDF5, so that might be why Pandas is failing to read it in. I know I ran into that issue at first when trying to do the SQLite stuff.

[klingaard]

Yeah, the example HDF5 files written by this tester can't be loaded into python either (even though compression is disabled):
https://github.com/sparcians/map/tree/master/sparta/simdb/test/HDF5Database

I'll look into it. This is a part of Sparta that has been carried over from generation to generation, but barely used. I personally want to use it more (it's very powerful).

[klingaard]

Ok, I figured it out. You need to provide the key to the first table in the HDF5 file. As an example, I got this to work with the SIMDB tester:

>>> import pandas as pd
>>> pd.read_hdf('00d00847-ba01-4724-8aa7-4795e6a07360.h5', key='MyCompound')
   ch   i1  ui1     i2    ui2         i4         ui4                   i8                  ui8          flt         dbl
0  -9   89   73   6339  22863  868619902  1575260798  3007835716998817119  4132581814464086664 -1236.011719 -193.799498
1 -62 -111  204  27822  61582  598558580  1399198217  1974705726194853838  5808522718447725331  -658.663147 -720.018602

You can figure out what the key is by doing this:

import h5py
h5py.File('00d00847-ba01-4724-8aa7-4795e6a07360.h5').keys()
<KeysViewHDF5 ['MyCompound']>

Hope this helps!

[danbone]

Thank you that works

[Deleted user]

Issue resolved.