Error: SyntaxError: "[object Object]" is not valid JSON when using io-ts and polyfact libraries

[kevin-btc]

In my TypeScript project, I'm using io-ts (v2.2.20) and polyfact (v0.1.7) on a macOS environment with Node.js (v19.4.0). When executing the generateWithTypeWithTokenUsage function with a prompt and io-ts type as parameters, I encounter the following error message:

An unknown error occurred while parsing the file file test/: SyntaxError: "[object Object]" is not valid JSON
Retrying generation... (1/3)
An unknown error occurred while parsing the file file test/: SyntaxError: "[object Object]" is not valid JSON
...
Error: An unknown error occured
    at new GenerationError (/Users/kev/Projects/ai-docs-package/node_modules/polyfact/dist/index.js:119:32)
    ...
    at processTicksAndRejections (node:internal/process/task_queues:95:5) {
  errorType: 'unknown_error'
}

Steps to Reproduce:

1. Prepare a prompt and io-ts type.
   the prompt in my case :

function formatReferencePrompt(content: string, path: string): string {
  return `
    Examine the provided content from file ${path} :
    \`\`\`\n ${content}\n\`\`\`
    
    Write a JSON object that describes that will be used to generate the reference of the file.
    The string inside the JSON must be plain text, and not contain any markdown or HTML.
    The JSON object should follow this type:
    \`\`\`
    {
        "description": string, // A 200 word description of the file and what it's used for.
        "references": [
            {
                "name": string, // The name of the function/method/class/type/structure/etc...
                "prototype": string, // The prototype of the function or method. Should only be used for functions and methods.
                "enum": [string], // The possible values of the enum. Should only be used for enums.
                "description": string, // A 100 word description of its purpose.
                "keywords": string[], // A list of keywords that can be used to search for this reference.
                "parameters": [
                    {
                        "name": string,
                        "type": string,
                        "description": string, // A description of the purpose of the parameter.
                    }
                ],
                "returns": {
                    "type": string,
                    "description": string, // A description of the purpose of the return value.
                }
                "errors": [
                    {
                        "name": string,
                        "description": string,
                    }
                ],
            }
        ],
        "examples": [
            {
                "title": string,
                "description": string,
                "example": string, // A commented code example that uses the functions in the reference. If the examples are bash command, make it so it's runnable from the root of the project and use the most standard way of running it. (This file path: \`root/${path}\`)
            }
        ]
    }
    \`\`\`
    Write as much example code as you can, and make sure to include comments that explain what the code does.
    Include all the possible errors that can occur.
    Only add the references of things that are defined in the file. Don't include imports or dependencies.
    Don't forget to make it so all the examples are runnable from the root of the project (This file path: \`root/${path}\`). YOU SHOULD NOT ASSUME THE USER IS ANYWHERE ELSE THAN THE ROOT DIRECTORY.
    To launch from the root directory, if the project contains multiple modules (which is clear by the fact that the lib/src path is not directly a child of the root directory), use the most standard way of running it. For example, in python you should use \`python -m <module>\`, in rust you should use \`cargo run --bin <bin>\`, etc...
    If the project doesn't appear to use modules and the code is directly in the root directory, don't use the module/bin.
    For example, a file located at \`root/src/main.rs\` should be runnable with \`cargo run\` from the root directory while a file located at \`root/module-a/src/main.rs\` should be runnable with \`cargo run --bin module-a\` from the root directory.
    If some command line arguments are defined in the file, don't forget to include them in the example. If the command line tool is not clear, replace it with <base_command>. (e.g. \`<base_command> --arg1 path1\`)
    The example should contain the command and command lines arguments only if the file is clearly defining a cli command or argument. If it is clearly related to CLI, don't forget to include the command line usage in the example as a bash program. You can use <base_command> is the base command is not defined in the file.
    ALL THE CLI COMMANDS USAGE DEFINED SHOULD BE EXPLICITLY WRITTEN IN THE EXAMPLES IN A COMMENTED BASH PROGRAM.
    
    Please only provide the JSON in a single json markdown code block with the keys described above. Do not include any other text.
    If the content is not code or doesn't define anything, just return "None" and nothing else.
    If at least one function is defined, you're not allowed to return None.
    You must include all the functions defined in the reference.
    Please make sure the JSON is a single line and does not contain any newlines outside of the strings.
    `;
}

the type :

const ReferenceParameter = t.type({
  name: t.string,
  type: t.string,
  description: t.string,
});

const ErrorType = t.type({
  name: t.string,
  description: t.string,
});

const ReturnsType = t.type({
  type: t.string,
  description: t.string,
});

const Example = t.type({
  title: t.string,
  description: t.string,
  example: t.string,
});

const TReference = t.type({
  name: t.string,
  prototype: t.string,
  enum: t.array(t.string),
  description: t.string,
  keywords: t.array(t.string),
  parameters: t.array(ReferenceParameter),
  returns: ReturnsType,
  errors: t.array(ErrorType),
});

const ReferenceType = t.type({
  description: t.string,
  references: t.array(TReference),
  examples: t.array(Example),
});

2. Call the generateWithTypeWithTokenUsage function with these parameters.
3. See the error.

Expected Behavior:

I expected the generateWithTypeWithTokenUsage function to return the completion of my prompt.

Actual Behavior:

However, I'm getting a SyntaxError with "[object Object]" is not valid JSON message.

Code Snippet:

Here's the snippet of the code where the error is generated:

console.info(`Generating reference for : ${file.path}`);

let prompt = formatReferencePrompt(content, path);

let updatedFile = { ...file };

for (let attempt = 0; attempt < 3; attempt++) {
  let reference;

  try {
    reference = String(
      await generateWithTypeWithTokenUsage(prompt, ReferenceType)
    );
  } catch (error) {
    console.log(ReferenceType, error);

    updateFileProgress(updatedFile);

    return updatedFile;
  }
}

Please let me know if there's any other information needed to investigate this error. Any assistance would be greatly appreciated.

[lowczarc]

I think the [object Object] part is just due to the use of String() instead of JSON.stringify() in your code. Doesn't seems to be a bug in the SDK.

Something relevant but out of the scope of this issue though: to reproduce it I had to remove the type part, since the Should only be used for enums. part confuses the model and the API doesn't support Optional types yet.
And the fact that you even had to specify a type in the first place demonstrate we need to change the way types are given to the SDK.
I've opened 2 other issues for this #4 and #5
I'll close this one.