esheldon/fitsio

Issue with slicing

matteobachetti opened this issue · 5 comments

matteobachetti commented

Hello, thanks for this great library!
For an application I'm doing, I need to quickly access slices of data from large FITS files, and I'm testing fitsio against astropy.io.fits. However, I'm having a problem when doing that, as follows.
In the MWE below, when I select a single row of the file, data are loaded correctly. If I use a slice, however, data get mangled in a strange way. I guess this might have something to do with the data types? Any help is appreciated

In [3]: import fitsio
   ...: fname = "ixpe02001099_det1_evt2_v02_bary_de200.fits"
   ...: fits = fitsio.FITS(fname)["EVENTS"]
   ...: print("First three single lines")
   ...: print(fits[0])
   ...: print(fits[1])
   ...: print(fits[2])
   ...: print("Now the three lines as a slice:")
   ...: print(fits[:3])
First three single lines
(7719, 1.93797604e+08, [   0,    0,    0,    0,    0,    0,    0,   67,   62,   24, -125,    2,   67, -109,   83,   -1], [  67, -113,  -24,  -52,  -65,  -13,   36, -108,  -95, -122,   81,  103,  -65,   -8,  -82, -128], -862093929, 0., 0., 0., 0., 0.)
(7720, 1.93797604e+08, [   0,    0,    0,    0,    0,    0,    0,  126,   63,   53,   64,   -6,   67, -102, -100,  -11], [  67, -108,   94,  -48,  -65,  -59,  -44,   61, -116,   97,   62,  -88,  -65,   -2,  -68,  120], 1864312268, 0., 0., 0., 0., 0.)
(7721, 1.93797604e+08, [   0,    0,    0,    0,    0,    0,    0,   63,   62,  -89,  -88,    8,   67, -106,  -56, -101], [  67, -105,   86,  -32,   63,  -10,   91,   44,   41,   99,  121, -100,   63,  -10,  -32,  -99], -224014269, 0., 0., 0., 0., 0.)
Now the three lines as a slice:
[(      7719,  1.93797604e+08, [   0,    0,    0,    0,    0,    0,    0,   67,   62,   24, -125,    2,   67, -109,   83,   -1], [  67, -113,  -24,  -52,  -65,  -13,   36, -108,  -95, -122,   81,  103,  -65,   -8,  -82, -128], -862093929, 1.081802e-41, 20.887808 , -4.8078172e+04, 6.22522714e-322, 0.00032431)
 (1133797072, -1.70539564e-01, [ -65,   -2,  -68,  120,  111,   31,   37,  -52,    0,    0,   30,   41,   65,  -89,   26,   59], [ -57,   64,  -53,   48,    0,    0,    0,    0,    0,    0,    0,   63,   62,  -89,  -88,    8], 1133955227, 3.026787e+02,  1.9246573,  5.0509605e-14, 1.42983813e+000, 0.        )
 (         0,  0.00000000e+00, [   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0], [   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0],          0, 0.000000e+00,  0.       ,  0.0000000e+00, 0.00000000e+000, 0.        )]

If I print(fits), it gives the following information and data types:

  file: ixpe02001099_det1_evt2_v02_bary_de200.fits
  extension: 1
  type: BINARY_TBL
  extname: EVENTS
  rows: 11165013
  column info:
    TRG_ID              i4  
    TIME                f8  
    STATUS              i1  array[16]
    STATUS2             i1  array[16]
    PI                  i4  
    W_MOM               f4  
    X                   f4  
    Y                   f4  
    Q                   f8  
    U                   f8

EDIT: a workaround for me is using read(rows=list_of_rows, columns=["TIME", "PI"])

esheldon commented

Could you provide a web link to the file so I can run some tests?

esheldon commented

That has access restrictions, I sent a request for access from my gmail address

  • 👍1
esheldon commented

I've got a PR to fix this: #409

Would you like to see if it works for you?

matteobachetti commented

It works! Thanks @esheldon for the super quick help!

In [5]: import fitsio
   ...: fname = "ixpe02001099_det1_evt2_v02_bary_de200.fits"
   ...: fits = fitsio.FITS(fname)["EVENTS"]
   ...: print("First three single lines")
   ...: print(fits[0])
   ...: print(fits[1])
   ...: print(fits[2])
   ...: print("Now the three lines as a slice:")
   ...: print(fits[:3])
   ...: 
   ...: 
First three single lines
(7719, 1.93797604e+08, [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], 67, 0.14893726, 294.65622, 287.81873, -1.19643081, -1.5426033)
(7720, 1.93797604e+08, [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], 126, 0.7080227, 309.22623, 296.74072, -0.17053956, -1.92101329)
(7721, 1.93797604e+08, [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], 63, 0.32745385, 301.56723, 302.6787, 1.39725891, 1.42983813)
Now the three lines as a slice:
[(7719, 1.93797604e+08, [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False],  67, 0.14893726, 294.65622, 287.81873, -1.19643081, -1.5426033 )
 (7720, 1.93797604e+08, [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], 126, 0.7080227 , 309.22623, 296.74072, -0.17053956, -1.92101329)
 (7721, 1.93797604e+08, [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], [False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False],  63, 0.32745385, 301.56723, 302.6787 ,  1.39725891,  1.42983813)]