Open big raw, analyze and nifti files fail
Closed this issue · 29 comments
Hi everyone,
When trying to load a large raw file, I've got very strange behaviour, only the n first slice are loaded, the rest is empty (0 value). All is fine with smaller files.
I tried to get the minimal code producing error, but I obtained something different: at a certain image size, the file is not load at all (16GB for me), see http://pastebin.com/VxEfe3KF
Do you have any idea ?
Hello. That looks like an issue with the maximal length of file offset.
Are you using CImg on Windows ? The sizeof(long) is only 32bits there, maybe I've used only long int to encode the file offsets. I'll check ASAP.
Thanks for reporting.
Thanks for your reply. I'm under linux.
ah. Strange, then... Is that a 32bits system ?
No, 64bits
Ok. I'll do some tests here (I'm working on Linux 64bits as well), and tell you what I get.
Ok, I hope you will have the same behaviour.
Note: I have also loading issue (segmentation fault) with large analyze (.img) format image, as well as with Nifti format, while it's working perfectly with smaller files.
See you,
PS: I'm actually stuck in loading large images (>10gb) with CImg, that s why I'm testing different formats, converting my images with Fiji. What would you suggest ?
It should be fixed now. I was using only unsigned int variables to store file offsets, so that was not good for files >4Gb.
Could you get the latest version of CImg.h and tell me if it's OK now ?
Thanks.
Yes it works for large RAW file now ! Thank you !
However, there is strange behaviour with large nifti and img format files ! may be the same error ?
I'll try to look at those issues too. But I don't have large nifti files so if I can't generate them, I won't be able to help. We'll see.
OK thanks a lot.
Note that FIJI can generate large nifti, Analyze [...] files format (http://fiji.sc/)
long is only guaranteed to be at least 32 bits.
https://en.wikipedia.org/wiki/C_data_types
The cstdint header should be used for fixed-size integers, and std::streampos should be used to hold file size or offsets
That's right. I'm porting most of the CImg code using 'long / unsigned long' to use 'size_t' and 'long64' instead, so that big images should be better managed on Windows in the future.
It shouldn't change anything on Linux though. Here, I've just fixed some old code in CImg<T>::load_raw() which was using unsigned int for file offsets.
Ok, It has may be nothing to do with, but I've a little problem since the last update of Cimg.h. I can't compile anymore my project and have several error for each file where I included cimg.h :
"multiple definitions of « cimg_library::cimg::fseek(IO_FILE, long, int)"
"multiple definitions of « cimg_library::cimg::ftell(IO_FILE) »"
It still work fine with an older version...
I made something wrong ?
Yes, I've committed at least one time an error. Try getting the latest version one more time, it should be fixed. I'm working on this file right now, and it may happen I break something for s short period of time, sorry :)
Should be OK now.
I've also fixed a similar problem in CImg<T>::load_analyze(), so maybe it's worth to try again with the latest version :)
No more segmentation fault for load_analyze() format, but same behaviour than with the old load_raw(), only the n-first slices are loaded.
And I can compile, thank you :)
Argh, I've probably done something wrong :)
I'll investigate and fix this ASAP.
Should be OK now :)
Let me know !
Yes, it works with large .img and .nifti now, good job !
(should I try with all supported format ? ;)
Any bug is good to catch, so if you have time to do some tests, yes sure !
Ok I will report bugs if I run into others :)
Thanks for your help, I can close the thread I think.
Hi,
It seems that Analyze and Nifti large files still aren't correctly managed in the last (repo) version of Cimg. The opened images are empty (or partially empty). The written files have mis-ordered pixels.
Is it possible to fix these formats for large files ?
Regards
Hi @uxhub ,
I've just noticed something wrong in method CImg<T>::save_analyze(), that was causing issues with the re-opening of the corresponding files.
Commit a0c40a4 should fix this issue (at least).
If you still have problems with the opening of large .nii files, could you tell me how have you generated them ? (or where can I find some to reproduce the bug ?).
Thanks
Hi,
The problem still is. Opening is concerned (image is partially loaded).
Large nii files can be produced with Fiji (http://fiji.sc/) and the nifti plugin (https://imagej.nih.gov/ij/plugins/nifti.html) to put in the plugin folder of fiji.
In Fiji: file->new->image : for example set a 32bits 1500x2048x1500 image (the size I've tested ~18GB)
Put something in the image, for example: select the elliptical selection (second icon of the icon menu), draw a big ellipse, and inverse the selection (ctrl+shift+i, or edit->invert) and said yes to process the whole stack.
Then: file->save as->Nifti-1
Thanks
The strange thing is that if I produce a large .nii file with CImg, and try to load it afterwards, that works.
Is it possible you save a large file with CImg and try to load it with CImg and/or something else ?
I'll probably won't have time to install ImageJ just to test the loading/saving of large .nii files.
One question though : are you sure this was working before ? Is it a regression, or something that never worked with CImg ?
It could help to track the bug.
Thanks.
Note, I've done a small modification to CImg<T>::load_analyze() with the hope it could solve your problem (with commit dab69e4).
Could you test please and tell me ?
The small modification solved the problem ! I can read properly external created nifti files. Also, I can create, write and read a nifti created with cimg.
As far as I know, It worked before the update with a recent version on github I made.
Many thanks
Ah good news !
That has been done in a blind way, so more intuition than analysis, but if this works, this is perfect :)
Nice ! Since I'm working with large files, I will follow these functionalities ;)
I close (again) the issue.
Thank you again.