vanna-ai/vanna-flask

Request Too Large for GPT-4o

Opened this issue · 3 comments

avinashkurup commented

Error Code: 429

Error Message:

Request too large for gpt-4o on tokens per min (TPM): Limit 30000, Requested 342389. The input or output tokens must be reduced in order to run successfully.

Description:
When attempting to process a request with the GPT-4 model, we encountered an error indicating that the request exceeded the allowed tokens per minute (TPM) limit. The current TPM limit is set at 30,000, but the request required 342,389 tokens.

Steps to Reproduce:

  1. Post a question that can yield a large number of results(rows).
  2. Upon generating the SQL and resulting DataFrame, the UI displays: "Error: The server returned an error. See the server logs for more details."
  3. The error in the logs reads: 127.0.0.1 - - [13/Jun/2024 10:52:21] "GET /api/v0/generate_followup_questions?id=1733c148-e1f0-472e-ac6b-321c5a4eb2bd HTTP/1.1" 500 -
    along with a message indicating that the request exceeds the TPM limit.

Expected Behavior:
The model should either process the request if it is within the allowed limits or provide clearer guidance on how to adjust the input to comply with the token restrictions.

Actual Behavior:
The model returns an error message without processing the request, indicating that the input or output tokens must be reduced.

Impact:
This issue prevents the processing of large requests and impacts our ability to utilize the GPT-4 model for tasks requiring extensive input or output.

Additional Information:

  • TPM Limit: 30,000
  • Tokens Requested: 342,389

Please let us know if additional information is required to address this issue.

avinashkurup commented

Hi @zainhoda,
Is there any fix incorporated in the app, such as setting TPM or any other alternative?
Looking forward to your response!

avinashkurup commented

Hi @zainhoda,

Any resolution to this issue would be greatly appreciated.

Thank you!

sadiqueshannure commented

Any resolution to this issue?