guidance-ai/guidance

In any case unable to generate more than 1000 tokens

a21-mohit opened this issue · 6 comments

a21-mohit commented

The bug
In any case unable to generate more than 1000 tokens, also tried by removing stop='}\n]'.

To Reproduce

models.AzureOpenAIChat(
                version=os.environ["OPENAI_API_VERSION"],
                azure_endpoint=os.environ["OPENAI_API_BASE"],
                api_key=os.environ["OPENAI_API_KEY"],
                azure_deployment="gpt-35-turbo-custom",
                model="gpt-35-turbo",
                echo=False,
                max_streaming_tokens=2048
            )
with assistant():
            my_generation_guidance += gen("my_generation", temperature=0, stop='}\n]', max_tokens=2048)

System info (please complete the following information):

  • OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Ubuntu 22.04
  • Guidance Version (guidance.__version__): 0.1.14
a21-mohit commented

I recently upgraded to newer syntax till now I was using very old guidance with older syntax style. In older package there was no issue but after applying newer syntax there seems to be issue in generating more than 1000 tokens. Any help is appreciated.

Harsha-Nori commented

Hey, this looks like a clear bug on our end. We have a safe default of 1000 set at places in the codebase, but they should be overridden by your argument passed in the gen call. We'll take a look at it -- thanks for reporting this!

azadveersingh commented

I also face same Issue, any help ?

slundberg commented

@azadveersingh @a21-mohit Sorry for the unhelpful default value... :) the gen function has a max_tokens arg that is by default set to 1k, I just made a PR to increate that to 1e10 so it won't limit token's unless you ask it to. Until that is merged you can just pass max_tokens=100000 to the gen call.

(and let me know if that fixes it)

a21-mohit commented

I haven't tested it yet but It doesn't seem to me gen issue but more the max_streaming_tokens issue.

a21-mohit commented

As you can see in my original post both max_streaming_tokens and max_tokens in gen function are 2048 and still the generation stops at 1K tokens

  • 👍1