Data is corrupted when subsampling a large file (8Gb) on windows

Question

Data is corrupted when subsampling a large file (8Gb) on windows

DylanMuir opened this issue 8 years ago · 13 comments

Dear Doctor Muir,

I am having a problem displaying a sub-sampled version of a 8gb mapped tensor file with the "imagesc" fnction.
If i choose the x and y gap per frame to be equal or greater than 5, the image will be destroyed at the 234th frame, which is around 2gb of data.
It always happens at the 234th frame, regardless where i start displaying. For example if i start at frame 234 the image will be destroyed immediately.
Data type used was "single".
Trying to display all the frames with a sub-sampling of 4x4 or less will result in no errors at all.
The mex'ed version was used. Trying to use he non mex'ed version did not work.
I would be happy to get your insight into why this is happening.

Best regards,
Jerome

Answer 1 · 2017-02-02T15:14:32.000Z

Hi Jerome,

That’s troubling. Just to confirm, if you do this:

imagesc(mtData(:, :, 234));
imagesc(mtData(1:2:end, 1:2:end, 234));

there is no problem, but this:

imagesc(mtData(1:5:end, 1:5:end, 234));

doesn’t work?

Is there an error message, or is the data simply corrupted?

Cheers,
Dylan.

Answer 2 · 2017-02-02T15:16:08.000Z

Dear Doctor Dylan,

That is exactly what is happening, the data is corrupted after 2gb. Please refer to the attached images.

gap = 5;
x_range = 1:gap:size(datablock,2);
y_range = 1:gap:size(datablock,1);
z_range = 1:1:1400;

val = 234;
imagesc(squeeze(datablock(yrange,xrange,zrange(val))));

This is what it should look like.

Thanks for the fast response.

Best regards,
Jerome

Answer 3 · 2017-02-02T15:40:42.000Z

Hi Jerome,

I can't reproduce this with a 3Gb file in double format (1024 x 768 x 512 images x 8 bytes = 3 Gb).
I can subsample as I choose, with no corruption. Could you please make sure you're using the latest commit of MappedTensor, and also let me know the data class of the file you are using?

Thanks,
Dylan.

Answer 4 · 2017-02-04T21:58:46.000Z

Dear Dr. Muir,

we started using your framework a while ago to analyze long video files as already mentioned by Jerome.
The easy access to tensors compared to memmapfiles is very much appreciated.

Regarding your questions:
The corruption of data happens at least in the formats uint16, doube and single.
Using the different formats, the corruption occus exactly after 2048MB (for each format) which suggest 16-bit file addressing.

Like you suggested, we tested the latets commit from here as well with the same result. Additionally, the effect was reproduced on two PCs with different compilers.

From the observations, I would conclude that it is not dependent on the file size.

What Jerome did not mention:
The actual data in the tensor is not altered by this:
Trying to display the video content after displaying a subsampled version gives still good results.

We tried Windows 7 and Windows 8.1 64bit with Matlab 2015b 64bit.

Do you need more information?
Thanks again for the excellent framework!

Best regards,

Michael

Answer 5 · 2017-02-07T13:58:11.000Z

Hi @mp1988,

Thanks for the extra information. It may be a windows-specific issue.

Could you please recompile the mapped_tensor_shim.c file with DEBUG defined (at the top of the file), and send me the output from running these commands in matlab?

datablock(1:5:20, 1:5:20, end)
datablock(1:20, 1:20, end)

Thanks,
Dylan.

Answer 6 · 2017-02-08T14:29:22.000Z

Dear Dr. Muir,

here are the outputs you asked for.

Sincerely,
Jerome

pathname =

D:\MatlabBa\MichaelVideo\40Hz_DC

filename =

PG_Grasshopper3 GS3-U3-23S6M_15355972()_X1920_Y1200_BPP2_LA1_FPS10_TIME2017-01-18-16-57-49_001

mts: Called with command [open]
mts/cof: Opening file [D:\MatlabBa\MichaelVideo\40Hz_DC\PG_Grasshopper3 GS3-U3-23S6M_15355972()_X1920_Y1200_BPP2_LA1_FPS10_TIME2017-01-18-16-57-49_001.dat]
mts/cof: Opening file in read-write mode.
mts/cof: File handle [000007FEEFDF1630], [8] bytes
mts/cof: System format is [ieee-le]
mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEFDF1630]
mts/crc: Chunks:
[-103991295][1][2304000]
mts/crc: Unique indices:
[-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648]...
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22]...
mts/crc: Data size: [1920][1200][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [2304000] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [00000000166D0060] (offset 0)
mts/crc: Seeking to [-207982592]
mts/crc: Single-read chunk, [4608000] bytes
Eavaluation Object created.
mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEFDF1630]
mts/crc: Chunks:
[-103991295][5][4]
[-103981695][5][4]
[-103972095][5][4]
[-103962495][5][4]
mts/crc: Unique indices:
[-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648]
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16]
mts/crc: Data size: [4][4][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [16] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [000000001C033E20] (offset 0)
mts/crc: Seeking to [-207982592]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001C033E28] (offset 8)
mts/crc: Seeking to [-207963392]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001C033E30] (offset 16)
mts/crc: Seeking to [-207944192]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001C033E38] (offset 24)
mts/crc: Seeking to [-207924992]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes

ans =

4×4 uint16 matrix

24385 608 624 656
53250 53250 53250 53250
36866 36866 36866 36866
32770 32770 32770 32770

mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEFDF1630]
mts/crc: Chunks:
[-103991295][1][20]
[-103989375][1][20]
[-103987455][1][20]
[-103985535][1][20]
[-103983615][1][20]
[-103981695][1][20]
[-103979775][1][20]
[-103977855][1][20]
[-103975935][1][20]
[-103974015][1][20]
[-103972095][1][20]
[-103970175][1][20]
[-103968255][1][20]
[-103966335][1][20]
[-103964415][1][20]
[-103962495][1][20]
[-103960575][1][20]
[-103958655][1][20]
[-103956735][1][20]
[-103954815][1][20]
mts/crc: Unique indices:
[-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648][-2147483648]...
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22]...
mts/crc: Data size: [20][20][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [400] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [0000000036F0E920] (offset 0)
mts/crc: Seeking to [-207982592]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0E948] (offset 40)
mts/crc: Seeking to [-207978752]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0E970] (offset 80)
mts/crc: Seeking to [-207974912]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0E998] (offset 120)
mts/crc: Seeking to [-207971072]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0E9C0] (offset 160)
mts/crc: Seeking to [-207967232]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0E9E8] (offset 200)
mts/crc: Seeking to [-207963392]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EA10] (offset 240)
mts/crc: Seeking to [-207959552]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EA38] (offset 280)
mts/crc: Seeking to [-207955712]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EA60] (offset 320)
mts/crc: Seeking to [-207951872]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EA88] (offset 360)
mts/crc: Seeking to [-207948032]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EAB0] (offset 400)
mts/crc: Seeking to [-207944192]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EAD8] (offset 440)
mts/crc: Seeking to [-207940352]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EB00] (offset 480)
mts/crc: Seeking to [-207936512]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EB28] (offset 520)
mts/crc: Seeking to [-207932672]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EB50] (offset 560)
mts/crc: Seeking to [-207928832]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EB78] (offset 600)
mts/crc: Seeking to [-207924992]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EBA0] (offset 640)
mts/crc: Seeking to [-207921152]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EBC8] (offset 680)
mts/crc: Seeking to [-207917312]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EBF0] (offset 720)
mts/crc: Seeking to [-207913472]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [0000000036F0EC18] (offset 760)
mts/crc: Seeking to [-207909632]
mts/crc: Single-read chunk, [40] bytes

ans =

20×20 uint16 matrix

Columns 1 through 16

24385 640 41538 608 672 656 640 608 640 592 624 672 592 592 656 720 672 608 608 624 704 624 608 592 640 624 656 592 640 592 608 672 704 656 688 624 704 656 576 688 688 672 624 608 736 656 608 592 624 640 640 640 688 656
640 624 592 592 720 592 704 640 656 624 608 656 608 640
624 608 688 656 656 656 704 672 592 704 608 640 576 656
656 656 640 688 608 576 576 704 704 624 624 560 656 656
704 640 688 608 640 672 640 640 624 640 624 624 592 576
608 688 672 688 640 656 688 592 704 688 624 576 656 624
608 576 640 640 656 688 656 624 640 688 608 608 624 592
704 704 656 640 640 640 656 560 656 656 608 624 576 592
640 688 640 720 704 592 624 688 592 608 640 672 608 672
656 624 656 640 672 640 656 640 624 672 688 656 624 672
672 704 592 640 704 704 656 672 704 688 576 704 640 656
640 688 656 624 624 640 704 688 672 592 608 640 640 640
688 640 688 608 592 640 672 624 688 656 688 640 704 592
672 656 736 672 640 672 624 688 640 704 704 672 752 640
624 688 672 672 640 624 640 608 576 688 656 656 656 640
672 608 704 672 640 704 576 672 656 672 688 608 576 640
656 672 608 640 704 672 736 656 672 656 656 592 704 608
672 688 640 688 672 704 720 720 656 624 720 704 688 608
656 560 624 640 656 608 704 656 736 720 656 672 640 688
560 640 560 592 672 736 688 656 672 672 704 720 688 672

Columns 17 through 20

 640     624     656     656
 624     624     576     608
 624     608     608     640
 592     656     576     656
 640     672     592     656
 560     592     672     672
 656     688     656     576
 640     560     704     560
 592     608     624     672
 608     672     720     656
 624     640     704     608
 624     656     576     560
 672     640     640     656
 560     672     672     608
 688     576     688     624
 656     656     720     656
 624     672     688     688
 624     704     672     656


 640     640     624     656
 656     608     640     656

Answer 7 · 2017-02-15T08:08:21.000Z

Hi @JeromeB211 @mp1988,

I have just pushed an update to a new branch iss/15. Please pull that branch, re-build mapped_tensor_shim.c with mex, and see if this commit fixes your problem. Please let me know either way.

There were a couple of lines where non-64 bit versions of fseek may have been used.

Best regards,
Dylan.

Answer 8 · 2017-02-15T14:51:04.000Z

Dear Dr. Muir,

Thank you very much for the update. Unfortunately it did not solve the problems we had. The data is even corrupted at the first frame now. Although the values of the corrupted Data changed, they are still not correct. I ran the same code of which i sent you the output last time. Here are the results now.
Thank you very much for your interest in our problem.

Sincerely ,
Jerome

pathname =

D:\MatlabBa\MichaelVideo\40Hz_DC

filename =

PG_Grasshopper3 GS3-U3-23S6M_15355972()_X1920_Y1200_BPP2_LA1_FPS10_TIME2017-01-18-16-57-49_001

mts: Called with command [open]
mts/cof: Opening file [D:\MatlabBa\MichaelVideo\40Hz_DC\PG_Grasshopper3 GS3-U3-23S6M_15355972()_X1920_Y1200_BPP2_LA1_FPS10_TIME2017-01-18-16-57-49_001.dat]
mts/cof: Opening file in read-write mode.
mts/cof: File handle [000007FEEB861630], [8] bytes
mts/cof: System format is [ieee-le]
Eavaluation Object created.
mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEB861630]
mts/crc: Chunks:
[4190976001][5][4]
[4190985601][5][4]
[4190995201][5][4]
[4191004801][5][4]
mts/crc: Unique indices:
[4190976001][4190976006][4190976011][4190976016][4190985601][4190985606][4190985611][4190985616][4190995201][4190995206][4190995211][4190995216][4191004801][4191004806][4191004811][4191004816]
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16]
mts/crc: Data size: [4][4][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [16] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [00000000173937E0] (offset 0)
mts/crc: Seeking to [8381952000]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [00000000173937E8] (offset 8)
mts/crc: Seeking to [8381971200]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [00000000173937F0] (offset 16)
mts/crc: Seeking to [8381990400]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [00000000173937F8] (offset 24)
mts/crc: Seeking to [8382009600]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEB861630]
mts/crc: Chunks:
[4190976001][5][4]
[4190985601][5][4]
[4190995201][5][4]
[4191004801][5][4]
mts/crc: Unique indices:
[4190976001][4190976006][4190976011][4190976016][4190985601][4190985606][4190985611][4190985616][4190995201][4190995206][4190995211][4190995216][4191004801][4191004806][4191004811][4191004816]
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16]
mts/crc: Data size: [4][4][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [16] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [000000001738AF60] (offset 0)
mts/crc: Seeking to [8381952000]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001738AF68] (offset 8)
mts/crc: Seeking to [8381971200]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001738AF70] (offset 16)
mts/crc: Seeking to [8381990400]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: ------- Reading chunk. UDP is [000000001738AF78] (offset 24)
mts/crc: Seeking to [8382009600]
mts/crc: Single-element skip-read: read [4] elements, [2] bytes per element, skip [8] bytes.
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes
mts/crc: Skipping [8] bytes

ans =

4×4 uint16 matrix

24385 608 624 656
640 608 624 576
640 608 624 576
640 608 624 576

mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEB861630]
mts/crc: Chunks:
[4190976001][1][20]
[4190977921][1][20]
[4190979841][1][20]
[4190981761][1][20]
[4190983681][1][20]
[4190985601][1][20]
[4190987521][1][20]
[4190989441][1][20]
[4190991361][1][20]
[4190993281][1][20]
[4190995201][1][20]
[4190997121][1][20]
[4190999041][1][20]
[4191000961][1][20]
[4191002881][1][20]
[4191004801][1][20]
[4191006721][1][20]
[4191008641][1][20]
[4191010561][1][20]
[4191012481][1][20]
mts/crc: Unique indices:
[4190976001][4190976002][4190976003][4190976004][4190976005][4190976006][4190976007][4190976008][4190976009][4190976010][4190976011][4190976012][4190976013][4190976014][4190976015][4190976016][4190976017][4190976018][4190976019][4190976020][4190977921][4190977922]...
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22]...
mts/crc: Data size: [20][20][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [400] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [000000013D84A140] (offset 0)
mts/crc: Seeking to [8381952000]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A168] (offset 40)
mts/crc: Seeking to [8381955840]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A190] (offset 80)
mts/crc: Seeking to [8381959680]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A1B8] (offset 120)
mts/crc: Seeking to [8381963520]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A1E0] (offset 160)
mts/crc: Seeking to [8381967360]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A208] (offset 200)
mts/crc: Seeking to [8381971200]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A230] (offset 240)
mts/crc: Seeking to [8381975040]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A258] (offset 280)
mts/crc: Seeking to [8381978880]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A280] (offset 320)
mts/crc: Seeking to [8381982720]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A2A8] (offset 360)
mts/crc: Seeking to [8381986560]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A2D0] (offset 400)
mts/crc: Seeking to [8381990400]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A2F8] (offset 440)
mts/crc: Seeking to [8381994240]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A320] (offset 480)
mts/crc: Seeking to [8381998080]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A348] (offset 520)
mts/crc: Seeking to [8382001920]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A370] (offset 560)
mts/crc: Seeking to [8382005760]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A398] (offset 600)
mts/crc: Seeking to [8382009600]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A3C0] (offset 640)
mts/crc: Seeking to [8382013440]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A3E8] (offset 680)
mts/crc: Seeking to [8382017280]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A410] (offset 720)
mts/crc: Seeking to [8382021120]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D84A438] (offset 760)
mts/crc: Seeking to [8382024960]
mts/crc: Single-read chunk, [40] bytes
mts: Called with command [read_chunks]
mts/crc: Reading chunks
mts/crc: File handle: [000007FEEB861630]
mts/crc: Chunks:
[4190976001][1][20]
[4190977921][1][20]
[4190979841][1][20]
[4190981761][1][20]
[4190983681][1][20]
[4190985601][1][20]
[4190987521][1][20]
[4190989441][1][20]
[4190991361][1][20]
[4190993281][1][20]
[4190995201][1][20]
[4190997121][1][20]
[4190999041][1][20]
[4191000961][1][20]
[4191002881][1][20]
[4191004801][1][20]
[4191006721][1][20]
[4191008641][1][20]
[4191010561][1][20]
[4191012481][1][20]
mts/crc: Unique indices:
[4190976001][4190976002][4190976003][4190976004][4190976005][4190976006][4190976007][4190976008][4190976009][4190976010][4190976011][4190976012][4190976013][4190976014][4190976015][4190976016][4190976017][4190976018][4190976019][4190976020][4190977921][4190977922]...
mts/crc: Reverse sort indices:
[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22]...
mts/crc: Data size: [20][20][1]
mts/crc: Data class: [uint16]
mts/crc: Header bytes: [0]
mts/crc: Data is big endian: [0]
mts/crc: System is little endian
mts/crc: Allocated read buffer: [400] elements, [2] bytes per elelemt
mts/crc: ------- Reading chunk. UDP is [000000013D8490A0] (offset 0)
mts/crc: Seeking to [8381952000]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8490C8] (offset 40)
mts/crc: Seeking to [8381955840]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8490F0] (offset 80)
mts/crc: Seeking to [8381959680]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849118] (offset 120)
mts/crc: Seeking to [8381963520]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849140] (offset 160)
mts/crc: Seeking to [8381967360]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849168] (offset 200)
mts/crc: Seeking to [8381971200]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849190] (offset 240)
mts/crc: Seeking to [8381975040]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8491B8] (offset 280)
mts/crc: Seeking to [8381978880]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8491E0] (offset 320)
mts/crc: Seeking to [8381982720]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849208] (offset 360)
mts/crc: Seeking to [8381986560]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849230] (offset 400)
mts/crc: Seeking to [8381990400]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849258] (offset 440)
mts/crc: Seeking to [8381994240]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849280] (offset 480)
mts/crc: Seeking to [8381998080]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8492A8] (offset 520)
mts/crc: Seeking to [8382001920]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8492D0] (offset 560)
mts/crc: Seeking to [8382005760]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D8492F8] (offset 600)
mts/crc: Seeking to [8382009600]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849320] (offset 640)
mts/crc: Seeking to [8382013440]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849348] (offset 680)
mts/crc: Seeking to [8382017280]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849370] (offset 720)
mts/crc: Seeking to [8382021120]
mts/crc: Single-read chunk, [40] bytes
mts/crc: ------- Reading chunk. UDP is [000000013D849398] (offset 760)
mts/crc: Seeking to [8382024960]
mts/crc: Single-read chunk, [40] bytes

ans =

20×20 uint16 matrix

Columns 1 through 16

24385 640 688 672 624 608 736 656 608 592 624 640 640 640 688 656
41538 608 640 624 592 592 720 592 704 640 656 624 608 656 608 640
672 656 624 608 688 656 656 656 704 672 592 704 608 640 576 656
640 608 656 656 640 688 608 576 576 704 704 624 624 560 656 656
640 592 704 640 688 608 640 672 640 640 624 640 624 624 592 576
624 672 608 688 672 688 640 656 688 592 704 688 624 576 656 624
592 592 608 576 640 640 656 688 656 624 640 688 608 608 624 592
656 720 704 704 656 640 640 640 656 560 656 656 608 624 576 592
672 608 640 688 640 720 704 592 624 688 592 608 640 672 608 672
608 624 656 624 656 640 672 640 656 640 624 672 688 656 624 672
704 624 672 704 592 640 704 704 656 672 704 688 576 704 640 656
608 592 640 688 656 624 624 640 704 688 672 592 608 640 640 640
640 624 688 640 688 608 592 640 672 624 688 656 688 640 704 592
656 592 672 656 736 672 640 672 624 688 640 704 704 672 752 640
640 592 624 688 672 672 640 624 640 608 576 688 656 656 656 640
608 672 672 608 704 672 640 704 576 672 656 672 688 608 576 640
704 656 656 672 608 640 704 672 736 656 672 656 656 592 704 608
688 624 672 688 640 688 672 704 720 720 656 624 720 704 688 608
704 656 656 560 624 640 656 608 704 656 736 720 656 672 640 688
576 688 560 640 560 592 672 736 688 656 672 672 704 720 688 672

Columns 17 through 20

 640     624     656     656
 624     624     576     608
 624     608     608     640
 592     656     576     656
 640     672     592     656
 560     592     672     672
 656     688     656     576
 640     560     704     560
 592     608     624     672
 608     672     720     656
 624     640     704     608
 624     656     576     560
 672     640     640     656
 560     672     672     608
 688     576     688     624
 656     656     720     656
 624     672     688     688
 624     704     672     656
 640     640     624     656
 656     608     640     656

Answer 9 · 2017-03-23T09:34:30.000Z

Hi,

Sorry it's taken me so long to get back to this. It's difficult to debug because I can't reproduce the problem.

When you say "the data is corrupted at the first frame", do you mean the first frame in the stack? Do you mean when skipping pixels (i.e. 1:5:20) or when reading all pixels (i.e. 1:20), or both?

Answer 10 · 2017-03-23T12:02:10.000Z

Hi, Here, I am referring to what I wrote last time: “ What works: Loading and displaying a video frame by frame . ->Works in branch iss/15 What does not work: 1) Introducing subsampling steps >4. 2) Accessing a time-series (stack) of single pixels behind the 2GB border (of the whole file). A =Mt(1,1,:) A (1,1, 10000) -> incorrect value B =Mt(1,1,10000) -> correct value “ I cannot say for sure what Jerome meant, but I think he meant the case skipping pixels (i.e. 1:5:20) /subsampling. Best regards Michael Von: Dylan Muir [mailto:notifications@github.com] Gesendet: Donnerstag, 23. März 2017 10:35 An: DylanMuir/MappedTensor <MappedTensor@noreply.github.com> Cc: Michael Paul <michael.paul1988@gmx.de>; Mention <mention@noreply.github.com> Betreff: Re: [DylanMuir/MappedTensor] Data is corrupted when subsampling a large file (8Gb) on windows (#15) Hi, Sorry it's taken me so long to get back to this. When you say "the data is corrupted at the first frame", do you mean the first frame in the stack? Do you mean when skipping pixels (i.e. 1:5:20) or when reading all pixels (i.e. 1:20), or both? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ_pvg1-gvMrECkPii0C6IfRM_m15tfMks5rojymgaJpZM4L1OsM> .

Answer 11 · 2017-03-24T10:02:21.000Z

Hi,

Please pull the latest commit to the iss/15 branch, recompile mapped_tensor_shim.c and try the following:

mt = MappedTensor(<file details>);
a = mt(1:20, 1, end);
b = mt(1:5:20, 1, end);
[a(1:5:end) b]

Answer 12 · 2017-03-24T13:10:42.000Z

Dear Dylan, I think you fixed it :) ( e = 0 ). gap = 5 a = video_sequence(1:20, 1, end); b = video_sequence(1:gap:20, 1, end); c =[a(1:gap:end) b]; d =a(1:gap:end); e =sum(d-b) I also tried to access the tensor video_sequence with gap =32 which worked fine, too. What did you have to change to make it work? Best regards, Michael Von: Dylan Muir [mailto:notifications@github.com] Gesendet: Freitag, 24. März 2017 11:02 An: DylanMuir/MappedTensor <MappedTensor@noreply.github.com> Cc: Michael Paul <michael.paul1988@gmx.de>; Mention <mention@noreply.github.com> Betreff: Re: [DylanMuir/MappedTensor] Data is corrupted when subsampling a large file (8Gb) on windows (#15) Hi, Please pull the latest commit to the iss/15 branch, recompile mapped_tensor_shim.c and try the following: mt = MappedTensor(<file details>); a = mt(1:20, 1, end); b = mt(1:5:20, 1, end); [a(1:5:end) b] — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ_pvpHnExVf4pxsKYz5BhNXBFKuQlX9ks5ro5SugaJpZM4L1OsM> .

Answer 13 · 2017-03-24T13:19:28.000Z

Hi Michael,

I'm glad this works for you now. I'll pull the changes into the master branch. There were a couple of issues performing file seeks beyond 2^32 bytes, and in mis-computing the file offsets when skipping over data.