Read terminates with Out of memory when using a 2GB window size
siara-cc opened this issue · 3 comments
siara-cc commented
Hi,
Thank you for making this wonderful port for ZStd!!
I am using following code:
using ZstdSharp;
byte[] ba = new byte[1000];
using var strm = System.IO.File.OpenRead(infile);
using var dstream = new DecompressionStream(strm, 131075, false);
dstream.SetParameter(ZstdSharp.Unsafe.ZSTD_dParameter.ZSTD_d_windowLogMax, 31);
Console.WriteLine("Here 1");
int iRead = dstream.Read(ba, 0, ba.Length);
Console.WriteLine("Here 2");
I am getting output:
Here 1
Out of memory.
If I don't do dstream.SetParameter()
, I am getting following error:
Here 1
Unhandled exception. ZstdSharp.ZstdException: Frame requires too much memory for decoding
at ZstdSharp.ThrowHelper.ThrowException(UIntPtr returnValue, String message)
at ZstdSharp.DecompressionStream.Read(Span`1 buffer)
at ZstdSharp.DecompressionStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at Reddit_Read_ZStd.Main(String[] args) in /Users/arun/git/Text_frequency_research/reddit_read_zstd.cs:line 104
Any idea how this can be solved? I have tried the same using Python and C++ versions of the Zstd library and it works fine.
oleg-st commented
Hi,
I am using AllocHGlobal(int)
which is limited to <2GB, need to use the nint
version also.
siara-cc commented
It is working! Thank a lot for quickly fixing this!!