Crashes on large conversation
IgnisDa opened this issue · 11 comments
Hi! Really, really impressive project. I love it!
I often hit this error:
It is non-deterministic (ofc) but I am happy to provide any more details you might need to debug this error.
I also suggest adding a --verbose flag so that the exact OpenAI response is returned and we can see what went wrong when running well.
Hi, @IgnisDa! Thank you for stopping by. And I appreciate the compliment.
I'm thinking of just adding retries in the next releases, but --verbose is a very nice addition, I'll need to think it through.
LMK when you make a release. I would like to test it.
It seems I get this error often in large files
The repo I am using: https://github.com/ignisda/ryot (The screenshot in the original issue is from a private repo).
Great pointer, thanks for investigating it. It is very probable that this issue is due to the chat history being too large: for now, the conversation does not get rolled up. I'll update you here once I do the rollups and the retries.
@IgnisDa, please see the latest update:
$ cargo install well
I would love to know if the issue still persists.
There seems to be a problem with the latest version:
archlinux@d024d0203361 /w/ryot (stats)> well 'what is the file apps/backend/src/miscellaneous/resolver.rs about?'
>> what is the file apps/backend/src/miscellaneous/resolver.rs about?
Error: Sent this: {
"messages": [
{
"content": "You are a command-line program that assists the user in querying and editing a codebase using a large language model. Your mission is to provide a conversational interface to perform tasks on the codebase. You have access to the repository through the file system. Use only the provided functions to navigate and manipulate the codebase.\nThe user will provide high-level instructions, and you will use your available functions to complete the tasks.\n\nAsk for clarification when needed, and keep responses concise, typically under a paragraph.\nMinimize explanations: you're an expert programmer talking to an expert programmer.\nIf unsure about an answer, request more information from the user.\n\nWhen asked about a particular definition, first use the `q` (query) function to find the files which have that definition. Then, use the `F` (read file) function to read them in detail and make sense of their contents.\n\nWhen asked about the whole codebase or cross-cutting concerns, start by identifying relevant files with the `q` (query) function.\nNext, use the `F` (read file) function to understand their contents.\nTo learn the file hierarchy, use the `f` (list files) function.\nTo understand the overall structure, read the `README.md` and CI files.\nThey will give you a hint of the overall structure.\n\nSpell out your intermediate thoughts for each folder visited, or whenever you see reasonable.\n\nWhen trying to edit the files, just provide the patch, without citing the full source.\nIf the tree is dirty, ask user's permissions before doing any edits.\n\nRemember, you've got this! Believe in your abilities and provide the best assistance possible.\n",
"role": "system"
},
{
"content": "what is the file apps/backend/src/miscellaneous/resolver.rs about?",
"role": "user"
}
],
"model": "gpt-3.5-turbo",
"tools": [
{
"function": {
"description": "query abstract syntax tree",
"name": "q",
"parameters": {
"properties": {
"path": {
"description": "relative path to the file to parse, or to a directory to parse all the files in",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}
},
"type": "function"
},
{
"function": {
"description": "list files",
"name": "f",
"parameters": {
"properties": {
"path": {
"description": "relative path to the directory to look into",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}
},
"type": "function"
},
{
"function": {
"description": "read file",
"name": "F",
"parameters": {
"properties": {
"path": {
"description": "relative path to the file to read",
"type": "string"
}
},
"required": [
"path"
],
"type": "object"
}
},
"type": "function"
},
{
"function": {
"description": "show commits log",
"name": "g",
"parameters": {}
},
"type": "function"
},
{
"function": {
"description": "show certain commit in details",
"name": "G",
"parameters": {
"properties": {
"hash": {
"description": "hash-like of the commit to show",
"type": "string"
}
},
"required": [
"hash"
],
"type": "object"
}
},
"type": "function"
},
{
"function": {
"description": "patch a file",
"name": "p",
"parameters": {
"properties": {
"patch": {
"description": "path to apply",
"type": "string"
},
"path": {
"description": "relative path to the file to read",
"type": "string"
}
},
"required": [
"path",
"patch"
],
"type": "object"
}
},
"type": "function"
}
]
}
Got this: {
"id": "chatcmpl-9xGo3CkjuTDb5PzRqkwBXGCTOadyi",
"object": "chat.completion",
"created": 1723912983,
"model": "gpt-3.5-turbo-0125",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_KjNtDNman92fJyef80SYhRFp",
"type": "function",
"function": {
"name": "q",
"arguments": "{\"path\":\"apps/backend/src/miscellaneous/resolver.rs\"}"
}
}
],
"refusal": null
},
"logprobs": null,
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 545,
"completion_tokens": 20,
"total_tokens": 565
},
"system_fingerprint": null
}
While Parsing: data did not match any variant of untagged enum CompletionResponse
Thanks for the quick reaction. The cause apparently was system_fingerprint being null. Feel free to give it another cargo install.
Works now! Thanks!
On an off note, do you know of a model that can handle a large number of input tokens?
I get this error with gpt 3.5
Perfect. Great to read that! 😌
Unsurprisingly, gpt-4o has the context window of 128k, which is almost eight times as high as gpt-3.5 you're using on the screenshot:
https://platform.openai.com/docs/models/gpt-4o.
Not supported by well, but still worth trying are:
- Claude (https://claude.ai/chat) with 200K or longer context,
- Google's Gemini (https://gemini.google.com/chat) with 1M+ context.
You will still have to copy and paste chunks around, but those still might work for larger projects.
Thanks for the help. I will use 4o.