oleg-st/ZstdSharp

Read terminates with Out of memory when using a 2GB window size

siara-cc opened this issue · 3 comments

Hi,

Thank you for making this wonderful port for ZStd!!

I am using following code:

using ZstdSharp;

byte[] ba = new byte[1000];
using var strm = System.IO.File.OpenRead(infile);
using var dstream = new DecompressionStream(strm, 131075, false);
dstream.SetParameter(ZstdSharp.Unsafe.ZSTD_dParameter.ZSTD_d_windowLogMax, 31);
Console.WriteLine("Here 1");
int iRead = dstream.Read(ba, 0, ba.Length);
Console.WriteLine("Here 2");

I am getting output:

Here 1
Out of memory.

If I don't do dstream.SetParameter(), I am getting following error:

Here 1
Unhandled exception. ZstdSharp.ZstdException: Frame requires too much memory for decoding
   at ZstdSharp.ThrowHelper.ThrowException(UIntPtr returnValue, String message)
   at ZstdSharp.DecompressionStream.Read(Span`1 buffer)
   at ZstdSharp.DecompressionStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Reddit_Read_ZStd.Main(String[] args) in /Users/arun/git/Text_frequency_research/reddit_read_zstd.cs:line 104

Any idea how this can be solved? I have tried the same using Python and C++ versions of the Zstd library and it works fine.

Hi,

I am using AllocHGlobal(int) which is limited to <2GB, need to use the nint version also.

Fixed in 0.6.3
@siara-cc Could you check it?

It is working! Thank a lot for quickly fixing this!!