SPHERE IFS observations

[joolof]

Hi,
I am having some issues trying to reduce SPHERE IFS observations. I have a cube with dimensions (39, 96, 290, 290) and when I try to run select_clean_data it returns an error message on the depth of the fits file.
Since SPHERE-IFS is described in get_infos_obs (with a fits file containing the wavelength information) I had assumed that I did not need to do anything to the fits file but I surely missed something. It looks like there is an extra dimension to the fits file that amical doesn't really like but I am not sure what I should do instead.

Below is what I am trying to do and the error message I am getting.
Thanks a bunch for providing such a useful tool!

datadir = "./"
file_t = os.path.join(datadir, "IFS_file.fits")
clean_param = {"isz": 149, "r1": 70, "dr": 2, "apod": True, "window": 65, "f_kernel": 3}
clip = True
cube_t = amical.select_clean_data(file_t, clip=clip, **clean_param, display=True)

and the error message is the following:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/.local/lib/python3.8/site-packages/amical/tools.py:70, in crop_max(img, dim, offx, offy, filtmed, f)
     69 try:
---> 70     im_med = medfilt2d(img, f)
     71 except ValueError:

File ~/.local/lib/python3.8/site-packages/scipy/signal/signaltools.py:1863, in medfilt2d(input, kernel_size)
   1861         raise ValueError("Each element of kernel_size should be odd.")
-> 1863 return sigtools._medfilt2d(image, kernel_size)

ValueError: object too deep for desired array

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Input In [3], in <cell line: 3>()
      1 clean_param = {"isz": 149, "r1": 70, "dr": 2, "apod": True, "window": 65, "f_kernel": 3}
      2 clip = True
----> 3 cube_t = amical.select_clean_data(file_t, clip=clip, **clean_param, display=True)

File ~/.local/lib/python3.8/site-packages/amical/data_processing.py:598, in select_clean_data(filename, isz, r1, dr, edge, clip, bad_map, add_bad, offx, offy, clip_fact, apod, sky, window, darkfile, f_kernel, verbose, ihdu, display)
    595 if add_bad is None:
    596     add_bad = []
--> 598 cube_cleaned = clean_data(
    599     cube,
    600     isz=isz,
    601     r1=r1,
    602     edge=edge,
    603     bad_map=bad_map,
    604     add_bad=add_bad,
    605     dr=dr,
    606     sky=sky,
    607     apod=apod,
    608     window=window,
    609     f_kernel=f_kernel,
    610     offx=offx,
    611     offy=offy,
    612     darkfile=darkfile,
    613     verbose=verbose,
    614 )
    616 if cube_cleaned is None:
    617     return None

File ~/.local/lib/python3.8/site-packages/amical/data_processing.py:482, in clean_data(data, isz, r1, dr, edge, bad_map, add_bad, apod, offx, offy, sky, window, darkfile, f_kernel, verbose)
    479 img1 = _remove_dark(img1, darkfile=darkfile, verbose=verbose)
    481 if isz is not None:
--> 482     im_rec_max = crop_max(img1, isz, offx=offx, offy=offy, f=f_kernel)[0]
    483 else:
    484     im_rec_max = img1.copy()

File ~/.local/lib/python3.8/site-packages/amical/tools.py:73, in crop_max(img, dim, offx, offy, filtmed, f)
     71     except ValueError:
     72         img = img.astype(float)
---> 73         im_med = medfilt2d(img, f)
     74 else:
     75     im_med = img.copy()

File ~/.local/lib/python3.8/site-packages/scipy/signal/signaltools.py:1863, in medfilt2d(input, kernel_size)
   1860     if (size % 2) != 1:
   1861         raise ValueError("Each element of kernel_size should be odd.")
-> 1863 return sigtools._medfilt2d(image, kernel_size)

ValueError: object too deep for desired array

[neutrinoceros]

Hi, could you share the fits file you are trying to clean here ? It would help a lot to work on a solution.

[joolof]

Hi, thanks for the answer.
Here is a link to the fits file (approx 1GB though ... sorry about that):
https://drive.google.com/file/d/1d1WryqLHOi9oAyIG2LId2i45bSJ2NweT/view?usp=sharing

[neutrinoceros]

Thank you !
@DrSoulain will take care of this soon

[DrSoulain]

Hi @joolof, thanks for your comment. Actually, the IFS compatibility is not fully public yet. We developed the interface for the usual IFS products, but your dataset seems to have a different format. Is it a specific SPHERE pipeline output?

[joolof]

Hi @DrSoulain thanks for having a look, the fits file was produced by the SPHERE data center at IPAG and I did not manipulate it afterwards. Should I try to reshape the cube somehow?
Thanks!

[DrSoulain]

I need to check :) But as first guess, I think that everything is already there, it's just matter of data compatibility and appropriate keywords. You are officially one of the first AMICAL user with SPHERE-IFS, so it's very nice to have feedbacks!