Stackoverflow when compressing a file larger than 254MB
oren-boop opened this issue · 7 comments
When compressing a folder with a file larger than 254 MB (266,338,304 bytes) there's a stackoverflow exception. This is odd since the file originally came from another MCDF file (MSI), so it's not a limitation of the MCDF file format.
this is the repeating pattern in the stack:
OpenMcdf.dll!OpenMcdf.CompoundFile.OnSizeLimitReached() Unknown
OpenMcdf.dll!OpenMcdf.SectorCollection.DoCheckSizeLimitReached() Unknown
OpenMcdf.dll!OpenMcdf.CompoundFile.OnSizeLimitReached() Unknown
OpenMcdf.dll!OpenMcdf.SectorCollection.DoCheckSizeLimitReached() Unknown
OpenMcdf.dll!OpenMcdf.CompoundFile.OnSizeLimitReached() Unknown
OpenMcdf.dll!OpenMcdf.SectorCollection.DoCheckSizeLimitReached() Unknown
OpenMcdf.dll!OpenMcdf.CompoundFile.OnSizeLimitReached() Unknown
...
to repro this create a large file (I used a file generator and filled the file with a repeating text pattern), and use something like the following code:
`
CFStream stream = null;
try
{
//try to get the original stream
root.TryGetStream(originalName, out stream);
}
catch (Exception)
{
}
if (stream != null)
{
//replace existing version of the stream(file) with the new version
...
}
else
{
//add a new file
stream = root.AddStream(originalName);
using (Stream fileData = fi.OpenRead())
{
stream.CopyFrom(fileData); <----------stack overflow here
}
}
I would appreciate any help on this.
Oren
A little more data: the issue happens when attempting to allocate enough room for the FAT:
OpenMcdf.dll!OpenMcdf.CompoundFile.OnSizeLimitReached() Line 248 C#
OpenMcdf.dll!OpenMcdf.SectorCollection.DoCheckSizeLimitReached() Line 49 C#
OpenMcdf.dll!OpenMcdf.SectorCollection.Add(OpenMcdf.Sector item) Line 128 C#
OpenMcdf.dll!OpenMcdf.CompoundFile.AllocateDIFATSectorChain(System.Collections.Generic.List<OpenMcdf.Sector> FATsectorChain) Line 1198 C#
OpenMcdf.dll!OpenMcdf.CompoundFile.AllocateFATSectorChain(System.Collections.Generic.List<OpenMcdf.Sector> sectorChain) Line 1100 C#
OpenMcdf.dll!OpenMcdf.CompoundFile.AllocateSectorChain(System.Collections.Generic.List<OpenMcdf.Sector> sectorChain) Line 1041 C#
OpenMcdf.dll!OpenMcdf.CompoundFile.SetSectorChain(System.Collections.Generic.List<OpenMcdf.Sector> sectorChain) Line 1015 C#
OpenMcdf.dll!OpenMcdf.CompoundFile.SetStreamLength(OpenMcdf.CFItem cfItem, long length) Line 2180 C#
OpenMcdf.dll!OpenMcdf.CompoundFile.WriteData(OpenMcdf.CFItem cfItem, byte[] buffer, long position, int offset, int count) Line 2330 C#
OpenMcdf.dll!OpenMcdf.CompoundFile.WriteData(OpenMcdf.CFItem cfItem, long position, byte[] buffer) Line 2312 C#
OpenMcdf.dll!OpenMcdf.CompoundFile.WriteData(OpenMcdf.CFItem cfItem, byte[] buffer) Line 2357 C#
OpenMcdf.dll!OpenMcdf.CFStream.SetData(byte[] data) Line 51 C#
OpenMcdf.dll!OpenMcdf.CFStream.CopyFrom(System.IO.Stream input) Line 232 C#
OpenMcdfRunner.exe!ReSec.MachineSafe.Server.Utils.OpenMcdfUtils.AddFolder(System.IO.DirectoryInfo folder, OpenMcdf.CFStorage root) Line 425 C#
OpenMcdfRunner.exe!ReSec.MachineSafe.Server.Utils.OpenMcdfUtils.Compress(System.IO.DirectoryInfo sourceDir, System.IO.FileInfo targetDest, System.IO.FileInfo refFile) Line 169 C#
OpenMcdfRunner.exe!OpenMcdf.Program.Main(string[] args) Line 55 C#
I think there's something wrong with lock sector checking. I'm going to analyze it. Could you please tell me what compound versions (3 or 4) are the files you're dealing with?
Thank you for your report.
@oren-boop it should work correctly know.
I fixed an issue with range lock sector when initializing a new CF file.
Please, let me kindly know if latest version works correctly.
Best Regards and thank you for your report.
Federico
I tested the latest version and the issue with compressing large files is indeed fixed. However, there is a regression in the version (maybe unrelated to this specific fix) that causes an exception when extracting MSI archives.
The stack trace is:
OpenMcdf.dll!OpenMcdf.StreamView.Read(byte[] buffer, int offset, int count) Line 181 C#
OpenMcdf.dll!OpenMcdf.CompoundFile.ReadData(OpenMcdf.CFStream cFStream, long position, byte[] buffer, int offset, int count) Line 2426 C#
OpenMcdf.dll!OpenMcdf.CFStream.Read(byte[] buffer, long position, int count) Line 172 C#
OpenMcdfRunner.exe!ReSec.MachineSafe.Server.Utils.OpenMcdfUtils.Extract.AnonymousMethod__0(string path, OpenMcdf.CFStream stream) Line 222 C#
OpenMcdfRunner.exe!ReSec.MachineSafe.Server.Utils.OpenMcdfUtils.VisitChildren.AnonymousMethod__0(OpenMcdf.CFItem item) Line 270 C#
OpenMcdf.dll!OpenMcdf.CFStorage.VisitEntries.AnonymousMethod__0(RedBlackTree.IRBNode targetNode) Line 509 C#
OpenMcdf.dll!RedBlackTree.RBTree.DoVisitTreeNodes(System.Action<RedBlackTree.IRBNode> action, RedBlackTree.IRBNode walker) Line 518 C#
OpenMcdf.dll!RedBlackTree.RBTree.DoVisitTreeNodes(System.Action<RedBlackTree.IRBNode> action, RedBlackTree.IRBNode walker) Line 523 C#
OpenMcdf.dll!RedBlackTree.RBTree.DoVisitTreeNodes(System.Action<RedBlackTree.IRBNode> action, RedBlackTree.IRBNode walker) Line 523 C#
OpenMcdf.dll!RedBlackTree.RBTree.DoVisitTreeNodes(System.Action<RedBlackTree.IRBNode> action, RedBlackTree.IRBNode walker) Line 514 C#
OpenMcdf.dll!RedBlackTree.RBTree.VisitTreeNodes(System.Action<RedBlackTree.IRBNode> action) Line 507 C#
OpenMcdf.dll!OpenMcdf.CFStorage.VisitEntries(System.Action<OpenMcdf.CFItem> action, bool recursive) Line 519 C#
OpenMcdfRunner.exe!ReSec.MachineSafe.Server.Utils.OpenMcdfUtils.VisitChildren(OpenMcdf.CFStorage root, string rootPath, System.Action<string, OpenMcdf.CFStream> streamAction, System.Action<string, OpenMcdf.CFStorage> storageAction) Line 263 C#
OpenMcdfRunner.exe!ReSec.MachineSafe.Server.Utils.OpenMcdfUtils.Extract(System.IO.FileInfo source, System.IO.DirectoryInfo targetInfo, System.Action<System.IO.FileInfo> fileAction, bool ole) Line 205 C#
it happens when we extract almost every MSI. I would be happy to provide more information if needed. I have a very easy repro on my computer.
Thanks,
Oren
Thank you @oren-boop for information. Could you provide a public msi file (or another file type) failing? Could you please provide also exception information aside from stack trace? I've tried a specific test for visiting an msi file and it worked correctly for the extraction part so I think that failure could possibly happen in other operations. Here's the test code:
`
private static int i = 0;
private void DoAction(CFItem c)
{
try
{
if (c is CFStream)
{
using (FileStream fs = new FileStream("ZZZ" + i++ + ".data", FileMode.Create))
{
byte[] b = new byte[c.Size];
((CFStream)c).Read(b, 0, (int)c.Size);
fs.Write(b, 0, (int)c.Size);
fs.Flush();
fs.Close();
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
[TestMethod]
public void Test_VISIT_MSI_AND_EXTRACT_FIX_83()
{
const String FILE_NAME = "f4ab2be.msi";
try
{
CompoundFile cf = new CompoundFile(FILE_NAME);
cf.RootStorage.VisitEntries(DoAction, true);
cf.Close();
}
catch (Exception ex)
{
Assert.Fail(ex.Message);
}
}
`
I apologize for my incorrect data. I did some more testing and found the problem in my code. I sincerely apologize for wasting your time. Thanks for your timely response!
No problem.
Thank you @oren-boop