Serialization of responses

[sswatson]

There are lots of reasons why someone would want to serialize a query response:

1. Saving to disk. We do this on the docs team.
2. Message-passing situations (e.g., web workers, or webviews in VS Code).
3. Fetching query results on the backend and forwarding the results to the frontend.

Currently, response objects that include Arrow relations do not survive serialization and deserialization via JSON. Instead, one needs to use tableToIPC/tableFromIPC to serialize/deserialize arrow relations and also MetadataInfo.fromJsonString/MetadataInfo.fromJson to serialize/deserialize metadata. This isn't all that hard to do, but it is pretty hard to figure out, and it would be reasonable for those methods to be provided by the SDK.

For inspiration, here's what we do in the VS Code extension:

export type SerializedArrow = {
  relationId: string;
  metadata: string;
  tableBase64: string;
};

export function serializeResults(results: ArrowRelation[]): SerializedArrow[] {
  return results.map((result) => {
    const ipc = tableToIPC(result.table, 'stream');
    return {
      relationId: result.relationId,
      metadata: MetadataInfo.toJsonString({
        relations: [{ 
          relationId: result.metadata,
          fileName: "",
        }]
      }),
      tableBase64: Buffer.from(ipc).toString('base64'),
    }
  });
}

export function deserializeResults(data: {
  relationId: string, metadata: string, tableBase64: string
}[]) {
  return data.map((row) => {
    const ipc = new Uint8Array(Buffer.from(row.tableBase64, 'base64'));
    const table = tableFromIPC(ipc);
    const metadataInfo = MetadataInfo.fromJson(JSON.parse(row.metadata));
    const { relations } = metadataInfo;
    const [ relation ] = relations;
    return {
      relationId: row.relationId,
      metadata: relation.relationId as RelationId,
      table,
    }
  });
}

[billscheidel-rai]

Note: This issue has been migrated to https://relationalai.atlassian.net/browse/RAI-7745.

This link is only accessible to employees of RelationalAI.