Serializing and Deserializing do not work as expected

[jwyuen]

Hello!

I'm attempting to serialize a DetachedTimestamp object after stamping data so that I can send it to a database for storage. When retrieving this stored data, deserializing the data either returns error or doesn't return the data as I would expect. In simplified terms:

OpenTimestamps.stamp(detachedTimestamp, options).then(() => {

          const serializedTimestamp = detachedTimestamp.serializeToBytes();

          // This will return an error
          const detachedOts = OpenTimestamps.DetachedTimestampFile.deserialize(serializedTimestamp);

          // This will return a valid DetachedTimestampFile but it will fail verification below
          const detachedOts = OpenTimestamps.DetachedTimestampFile.fromBytes(new OpenTimestamps.Ops.OpSHA256(), serializedTimestamp);

          console.log(detachedTimestamp);
          console.log(detachedOts);
          infoResult = OpenTimestamps.info(detachedTimestamp);
          console.log(infoResult);
          infoResult = OpenTimestamps.info(detachedOts);
          console.log(infoResult);


          const whitelist = new UrlWhitelist([
            'https://*.calendar.opentimestamps.org', 
            // 'https://*.calendar.eternitywall.com', 
            'https://*.calendar.catallaxy.com', 
          ]);

          const options = {
            timeout: 2,
            insight: {
              chain: 'bitcoin',
              urls: [
                'https://www.localbitcoinschain.com/api',
                'https://blockexplorer.com/api',
                // 'https://search.bitaccess.co/insight-api',
                // 'https://insight.bitpay.com/api',
              ],
            },
            whitelist,
          };

          OpenTimestamps.verify(detachedOts, detachedTimestamp, options).then((verifyResult) => {
            console.log(verifyResult);
          })
          .catch((err) => {
            console.error(err);
          });

}); 

Is my usage or assumption of what the serializeToBytes(), deserialize(), and fromBytes() functions incorrect, or have I encountered a bug here?

[lvaccaro]

Hi, DetachedTimestampFile object is the virtual representation of a proof in OpenTimestamps protocol. The proof is a merkle tree of data where the original hash is the head and attestations are the tips.
I try to explain better the main methods:

• serializeToBytes: serialize the memory object timestamp to a bytearray (= binary ots proof file)

const fileOts = detached.serializeToBytes();

• deserialize: deserialize the ots proof to memory object timestamp

const detached = OpenTimestamps.DetachedTimestampFile.deserialize(fileOts);

• fromBytes: initialize a new memory object timestamp from an byte array or buffer (this will be hashing with the hashing function defined in the 1st parameter)

const detached = OpenTimestamps.DetachedTimestampFile.fromBytes(new OpenTimestamps.Ops.OpSHA256(), file);

• fromHash: initialize a new memory object timestamp from an hash (in hex format)

const detached = OpenTimestamps.DetachedTimestampFile.fromHash(new OpenTimestamps.Ops.OpSHA256(), hash);

Note:

• serialization error: which error? could you provide options and a full test? Check with the working example written below.
• fromBytes: doesn't deserialize the timestamp, but build a new one from a byte array
• UrlWhitelist: now it is not exposed in the library interface
• timeout: should be put inside insight group & 2sec are too few for liteVerification
• verify function: return list of verified attestations indexed by chain, verify declaration; in your case, you have no verified attestations because you timestamped few lines before: stamp() returns a promise of ots calendar to timestamp (called PendingAttestion attestation). In your info output you could read the merkle tree and all the verifications (check you have no verified attestations).

...
verify PendingAttestation('https://alice.btc.calendar.opentimestamps.org')

Working example:

const OpenTimestamps = require('javascript-opentimestamps');
const file = Buffer.from('5468652074696d657374616d70206f6e20746869732066696c6520697320696e636f6d706c6574652c20616e642063616e2062652075706772616465642e0a','hex');
const detachedTimestamp = OpenTimestamps.DetachedTimestampFile.fromBytes(new OpenTimestamps.Ops.OpSHA256(), file);
OpenTimestamps.stamp(detachedTimestamp).then( ()=>{
	const serializedTimestamp = detachedTimestamp.serializeToBytes();
	const detachedOts = OpenTimestamps.DetachedTimestampFile.deserialize(serializedTimestamp);
         // note that `detachedTimestamp` is equals to `detachedOts`
	console.log(detachedTimestamp);
        console.log(detachedOts);
        infoResult = OpenTimestamps.info(detachedTimestamp);
        console.log(infoResult);
        infoResult = OpenTimestamps.info(detachedOts);
        console.log(infoResult);

	const options = {
          insight: {
              timeout: 30,
              chain: 'bitcoin',
              urls: [
                'https://www.localbitcoinschain.com/api',
                'https://blockexplorer.com/api',
                // 'https://search.bitaccess.co/insight-api',
                // 'https://insight.bitpay.com/api',
              ],
            }
        };
	OpenTimestamps.verify(detachedOts, detachedTimestamp, options).then((verifyResult) => {
        	console.log(verifyResult);
        	// verify output is empty because no verified attestations found, wait to be upgraded
        })
        .catch((err) => {
        	console.error(err);
        });
}).catch(err => {
	console.error(err);
});

[lvaccaro]

I suggest to read the post JS OpenTimestamps by example.

[jwyuen]

Thanks for the clarification!

The error I'm seeing when I use deserialize is:

ops.js:77 Unknown operation tag:  00

Btw I included the verify function mostly for debugging since my original problem was that I was trying to figure out why my original timestamp and my timestamp created from deserialized byte array were different.

Anyways, it looks like if I can figure out why deserialize isn't working that would pretty much solve my problem.... Maybe the issue is that I'm running the code in a browser? More specifically I'm running it on a react + webpack setup on Chromium Version 68.0.3440.75 (Developer Build) built on Debian 9.5, running on Debian 9.5 (64-bit)

[lvaccaro]

What version of the library are you using? Can you attach your parameters?
OpenTimestamps.stamp(detachedTimestamp, options).then(() => {

Compare your parameters with this working example on JS
https://gist.github.com/lvaccaro/f9e0aa000227e3fcf84c6cdeb5b39fb1#file-ots_test_in_browser-js
or test directly on JSFiddle console.

[jwyuen]

Ya running your code works. It also looks like my code works once I remove options from stamp(). The options I was using that was causing the error are:

        const stampOptions = {
          calendars: [
            'https://*.calendar.opentimestamps.org', // Run by Peter Todd
            // 'https://*.calendar.eternitywall.com', // Run by Riccardo Casatta of Eternity Wall
            'https://*.calendar.catallaxy.com', // Run by Vincent Cloutier of Catallaxy
          ],
        };

[lvaccaro]

calendars could not be grouped (maybe you are confusing with whitelist):

• calendars: url of calendars to override default calendars (not grouped) used in stamp()
• whitelist: list of remote calendar to accept proofs (grouped as you defined) used in upgrade() and verify()

For a list of stamp() options check jsdoc.