Able to load a OneNote from alternative packaging format

Question

Able to load a OneNote from alternative packaging format

nddipiazza opened this issue 3 years ago · 17 comments

I am trying to parse a OneNote package using this interop test suite project.

This project is saved directly from SharePoint online thus it is stored in:

MS-ONESTORE
2.8 Alternative Encoding Using the File Synchronization via SOAP Over HTTP Protocol

With a recent #56 had been merged with the alternative packaging support added. So I can actually load this format using the programming sdk:

      byte[] oneNoteBytes = File.ReadAllBytes(@"testOneNoteMultiplePages.one");

      SharedContext.Current.IsMsFsshttpRequirementsCaptured = false;

      AlternativePackaging alternatePackageOneStoreFile = new AlternativePackaging();
      alternatePackageOneStoreFile .DoDeserializeFromByteArray(oneNoteBytes, 0);

My ultimate goal here is to be able to obtain the MS-ONESTORE packaging so that I can extract text from this document.

So I expected this to work:

      MSONESTOREParser onenoteParser = new MSONESTOREParser();
      MSOneStorePackage package = onenoteParser.Parse(alternatePackageOneStoreFile.dataElementPackage);

But this does not work it fails during parsing:

Unhandled exception. System.InvalidOperationException: Unexpected to meet the byte array end.
   at Microsoft.Protocols.TestSuites.Common.BitReader.GetBytes(Int32 needReadlength, Int32 size) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\Common\BitField.cs:line 217
   at Microsoft.Protocols.TestSuites.Common.BitReader.ReadUInt32(Int32 readingLength) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\Common\BitField.cs:line 121
   at Microsoft.Protocols.TestSuites.MS_ONESTORE.ObjectSpaceObjectStreamHeader.DoDeserializeFromByteArray(Byte[] byteArray, Int32 startIndex) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\Common\ONESTORE\OtherStructures\ObjectSpaceObjectStreamHeader.cs:line 50
   at Microsoft.Protocols.TestSuites.MS_ONESTORE.ObjectSpaceObjectStreamOfOSIDs.DoDeserializeFromByteArray(Byte[] byteArray, Int32 startIndex) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\Common\ONESTORE\OtherStructures\ObjectSpaceObjectStreamOfOSIDs.cs:line 44
   at Microsoft.Protocols.TestSuites.MS_ONESTORE.ObjectSpaceObjectPropSet.DoDeserializeFromByteArray(Byte[] byteArray, Int32 startIndex) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\Common\ONESTORE\OtherStructures\ObjectSpaceObjectPropSet.cs:line 46
   at Microsoft.Protocols.TestSuites.SharedAdapter.HeaderCell.CreateInstance(ObjectGroupDataElementData objectElement) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\SharedTestSuite\SharedAdapter\Stack\ONESTORE\MSOneStorePackage.cs:line 125
   at Microsoft.Protocols.TestSuites.SharedAdapter.MSONESTOREParser.ParseHeaderCell(RevisionManifestDataElementData headerCellRevisionManifest) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\SharedTestSuite\SharedAdapter\Helper\MSONESTOREParser.cs:line 99
   at Microsoft.Protocols.TestSuites.SharedAdapter.MSONESTOREParser.Parse(DataElementPackage dataElementPackage) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\SharedTestSuite\SharedAdapter\Helper\MSONESTOREParser.cs:line 53
   at TestFSSHTTPB.Program.Main(String[] args) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\MS-FSSHTTP-FSSHTTPB\TestFSSHTTPB\Program.cs:line 26

Here is the one note file i am testing with:
testOneNoteMultiplePages.zip

Is my understanding of the format flawed here? I assumed that I'd be able to parse the data element package within the alternate packaging format using the MS-OneStore parser.

When I was talking to Tom Jebo, he was mentioning that there may be a need to "combine" some data packaging elements to make this work. Is that the case here?

How do I use this C# library to obtain the MS one store package from the 2.8 alternative packaging format?

Answer 1 · 2021-10-26T16:43:11.000Z

@stone-li just checking to see if you saw this? was wondering i'm properly using your new AlternativePackaging object? Or if I'm totally off-base here?

Answer 2 · 2021-10-27T10:27:46.000Z

@nddipiazza Yes, we are working on it. And I am from Test Suite project team and will let you know once I have any updates.

Answer 3 · 2021-10-27T17:34:15.000Z

Awesome thank you!

Answer 4 · 2021-11-02T15:07:05.000Z

@nddipiazza do you have any experiences to call parser function in MSONESTOREParser successfully before? Maybe it is about section 2.7 Transmission by Using the File Synchronization via SOAP Over HTTP Protocol. So far we could repro your issue, but have not found where the problem is. I would appreciate it if you give me more background info.

Answer 5 · 2021-11-03T12:07:18.000Z

@ChangDu2021 Hi thanks for the response.

So the use case is I am trying to extracting text from the OneNote files saved from SharePoint Online.

The data appears to follow the "Alternative Packaging structure".

I am able to use this C# module to successfully serialize into the AlternativePackaging object.

I am trying to figure out how to use this C# module how to take this AlternativePackaging object and obtain the MSOneStorePackage that is stored within it.

Answer 6 · 2021-11-04T03:20:20.000Z

@nddipiazza I got your use case. I am wondering do you use Parser function in MSOneStorePackage successfully before? Now I am not sure whether MSOneStorePackage obtained from AlternativePackaging is not correct or Parser function is not correct? Now we are trying to verify if the problems is that Parser function doesn't work.

public MSOneStorePackage Parse(DataElementPackage dataElementPackage)

Answer 7 · 2021-11-06T01:07:52.000Z

@ChangDu2021 No I have not successfully used it before.

But this code is in the test suite:

FsshttpbResponse fsshttpbResponse = FsshttpbResponse.DeserializeResponseFromByteArray(subResponseBinary, 0);
MSONESTOREParser onenoteParser = new MSONESTOREParser();
MSOneStorePackage package = onenoteParser.Parse(fsshttpbResponse.DataElementPackage);

So we should be able to take the result of MS-FSSHTTPB and parse it into a MS one note file.

But you have now added the AlternativePackaging object. How do I parse this object into an MS-ONESTORE compliant object?

Answer 8 · 2021-11-08T09:43:08.000Z

@nddipiazza We have verified the result of MS-FSSHTTPB could be parsed successfully. There is an test case about it called MSONESTORE_S01_TC01_QueryOneFileContainsFileData(). So, we suspect that the data element package between 2.7 and 2.8 is different.

Answer 9 · 2021-11-08T13:45:54.000Z

Is there some way to convert 2.8 into a 2.7 package?

Answer 10 · 2021-11-11T10:14:30.000Z

@nddipiazza could you please use attached OneNote file to parser it into a MS-ONESTORE compliant object? At my endpoint, it works. The content in OneNote file is the same as what in your OneNote file. Maybe, the only difference is the way how to get it. I created it using my local OneNote app, and save it into a folder on OneDrive directly, then go to OneDrive site to download the folder. If it works, could you please follow this way to create the OneNote file?
TestSuite.zip

Answer 11 · 2021-11-17T08:11:59.000Z

@nddipiazza Could you please try getting the code under Master branch from below url to test your .one file? At our dev env, it works to parse the .one file saved from OneDrive successfully. And the changes have not been merged into OfficeDev/Interop-TestSuites. If it works at your side, could you please let me know?

https://github.com/xian-rong/Interop-TestSuites

Answer 12 · 2021-11-21T20:53:40.000Z

Sorry for my long delay! Busy with my day job and haven't had time to respond.

I tried that branch, same error:

Unhandled exception. System.InvalidOperationException: Unexpected to meet the byte array end.
   at Microsoft.Protocols.TestSuites.Common.BitReader.GetBytes(Int32 needReadlength, Int32 size) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\Common\BitField.cs:line 217
   at Microsoft.Protocols.TestSuites.Common.BitReader.ReadUInt32(Int32 readingLength) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\Common\BitField.cs:line 121
   at Microsoft.Protocols.TestSuites.MS_ONESTORE.ObjectSpaceObjectStreamHeader.DoDeserializeFromByteArray(Byte[] byteArray, Int32 startIndex) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\Common\ONESTORE\OtherStructures\ObjectSpaceObjectStreamHeader.cs:line 50
   at Microsoft.Protocols.TestSuites.MS_ONESTORE.ObjectSpaceObjectStreamOfOSIDs.DoDeserializeFromByteArray(Byte[] byteArray, Int32 startIndex) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\Common\ONESTORE\OtherStructures\ObjectSpaceObjectStreamOfOSIDs.cs:line 44
   at Microsoft.Protocols.TestSuites.MS_ONESTORE.ObjectSpaceObjectPropSet.DoDeserializeFromByteArray(Byte[] byteArray, Int32 startIndex) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\Common\ONESTORE\OtherStructures\ObjectSpaceObjectPropSet.cs:line 46
   at Microsoft.Protocols.TestSuites.SharedAdapter.HeaderCell.CreateInstance(ObjectGroupDataElementData objectElement) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\SharedTestSuite\SharedAdapter\Stack\ONESTORE\MSOneStorePackage.cs:line 125
   at Microsoft.Protocols.TestSuites.SharedAdapter.MSONESTOREParser.ParseHeaderCell(RevisionManifestDataElementData headerCellRevisionManifest) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\SharedTestSuite\SharedAdapter\Helper\MSONESTOREParser.cs:line 99
   at Microsoft.Protocols.TestSuites.SharedAdapter.MSONESTOREParser.Parse(DataElementPackage dataElementPackage) in C:\lucidworks\Interop-TestSuites\FileSyncandWOPI\Source\SharedTestSuite\SharedAdapter\Helper\MSONESTOREParser.cs:line 53
   at ConsoleApp2.Program.Main(String[] args) in C:\Users\nicho\source\repos\ConsoleApp2\ConsoleApp2\Program.cs:line 24

Steps to reproduce:

Log into any SharePoint Online Document Library and create a new One Note file:

Create a simple OneNote file with a single page "Test" with content "Test"

In Firefox, download the section .one file via:

/Shared%20Documents/$NotebookName/$SectionName.one

Example from above:

https://mytenant.sharepoint.com/Shared%20Documents/TestFile/Untitled Section.one

By using Firefox, it will download it as an attachment instead of trying to open OneNote, etc.

Download the file to disk:

Try to load this file via:

using Microsoft.Protocols.TestSuites.Common;
using Microsoft.Protocols.TestSuites.MS_ONESTORE;
using Microsoft.Protocols.TestSuites.SharedAdapter;
using System.IO;

namespace ConsoleApp2
{
  class Program
  {
    static void Main(string[] args)
    {
      byte[] oneNoteBytes = File.ReadAllBytes(@"Untitled Section.one");

      SharedContext.Current.IsMsFsshttpRequirementsCaptured = false;
      AlternativePackaging alternatePackageOneStoreFile = new AlternativePackaging();
      alternatePackageOneStoreFile.DoDeserializeFromByteArray(oneNoteBytes, 0);

      MSONESTOREParser onenoteParser = new MSONESTOREParser();
      MSOneStorePackage package = onenoteParser.Parse(alternatePackageOneStoreFile.dataElementPackage);
    }
  }
}

This will now reproduce the error:

Does that help clarify the issue?

Answer 13 · 2021-11-22T10:06:47.000Z

@nddipiazza There is a typo in the last pr, now it has been fixed. Could you please try again to run your code under below branch? Apologize for any inconvenience.
https://github.com/xian-rong/Interop-TestSuites

Answer 14 · 2021-11-22T14:32:04.000Z

It works now!! Sounds like you fixed it.

Answer 15 · 2021-11-23T02:40:00.000Z

@nddipiazza Sounds great. Thanks for your quick response.

Answer 16 · 2021-12-05T20:15:37.000Z

this issue was resolved

Answer 17 · 2021-12-05T23:48:41.000Z

FYI this was merged in: #65