Can't run the tiktokenizer from the app folder in next.js
gablabelle opened this issue ยท 2 comments
Hello,
Trying to run tiktokenizer from the app folder and I'm getting the following error:
Error: Element type is invalid. Received a promise that resolves to: [object Promise]. Lazy element type must resolve to a class or function.
Here is the branch to reproduce the error.
Hi @gablabelle! Thanks for the contribution. I've tried to use the new App Router as well, but it does seem like there's some upstream issue with React Server Components (RSC) and NextJS implementation which does not allow importing WASM modules / the WASM package of tiktoken
.
One possible solution would be to replace tiktoken
with js-tiktoken
, if that's needed, but I'm still keen on experimenting with WASM and RSC ๐
Hi @gablabelle! Thanks for the contribution. I've tried to use the new App Router as well, but it does seem like there's some upstream issue with React Server Components (RSC) and NextJS implementation which does not allow importing WASM modules / the WASM package of
tiktoken
.One possible solution would be to replace
tiktoken
withjs-tiktoken
, if that's needed, but I'm still keen on experimenting with WASM and RSC ๐
I've since migrated to js-tiktoken and it works! ๐๐ฝ