unable to generate quiz due to code
Opened this issue · 18 comments
I used Cohere to get an API key.
When I click generate quiz I get:
Expected ',' or '}' after property value in JSON at position 2365
Can you try again? I just tested it myself and both the Command R and Command R+ models worked on the first try.
If it still doesn't work after multiple attempts, can you provide the model you used, the token count of your notes, and the type and number of questions you tried to generate?
Hey there!
It worked now with the same short note that I tried it with in the beginning (this plugin is so amazing I love it!)
I tried another long note and I got Unexpected end of JSON input
How many tokens was the note and how many questions were you trying to generate?
The new one was 24088 tokens.
How many questions did you try to generate?
I clicked on "Generate Quiz" with the standard settings, so 7.
Ok, I'll look into it.
Hey, just a heads up: I still have this problem.
I did some testing and turns out Cohere's models return nothing if your prompt exceeds ~4000 tokens. Note that this number can vary depending on the content of your notes because I use an extremely simple tokenizer which doesn't perfectly reflect how actual models tokenize inputs.
I plan to fix this by breaking down large inputs into chunks and processing each one separately. But in the meantime, I recommend either keeping each generation request under 4000 tokens, or using Google Gemini which can easily handle tens of thousands of tokens.
Thanks!
So I tried to test it with Gemini just now and I got the same error...
That's surprising. I just tried with 80,000 tokens and had no issues. What generation language are you using and did your selected notes contain any "special" text (code, LaTeX, tables, non-English characters, embeds, etc.)?
I use german as the language for the questions to be generated in.
The only "special" text symbols I use are emojis, I guess.
But if I use different files (that have no emojis) I get the same error message (only with different "position")
I just tried again using German as the generation language and even translated my notes into German but didn't have any problems, so this is really strange. Are you using the default Google API base URL or a third-party API? That seems like the only factor left that hasn't been considered.
I have not changed anything about the default API...
I'm also really confused about it. Is there anything else I could give you to tackle the problem better? Like a bug-report or something? (I don't know how to get that tho).
Thank you for your efforts!
I've attached a custom build of the latest release which has some logging for Cohere and Gemini. Go to YourVault/.obsidian/plugins/quiz-generator and replace the main.js file there with the attached one. Then try generating again with the Developer Tools open in Obsidian by doing Ctrl + Shift + I on Windows or Command + Option + I on Mac.
Edit: For some reason Windows Defender flags it as a virus when I tried downloading it even though I made the file so let me know if you can't download it.
Edit2: I found a workaround by replacing the zip file with a .txt file. You can just follow the instructions above but make sure you change the file extension from .txt to .js first.
Hey there!
Finally had the time to do this.
I did everything as you said.
Check out this log:
Quiz Generator is only in the last few lines
Thanks for the logs. I'm able to reproduce the bug with Cohere so I'll put out a fix for that soon.
Are you able to also send the logs when you try generating with Gemini? I can't get Gemini to fail no matter how many tokens I use so it'd be very helpful to see what error you're getting.
Hey there,
here is a log with Google:
obsidian.md-1731513334761.log
and another log with Cohere:
obsidian.md-1731513434176.log