When running 1.0.9 there is a Missing tiktoken_bg.wasm error
Closed this issue · 1 comments
jensen commented
TLDR; Hardcoded path in postprocess script causes module to not be found
I recently upgraded to 1.0.9 to include gpt 3.5 16k support.
I have switched to tiktoken
from @dqbd/tiktoken
imports.
When running my app which is built using esbuild. I see the following error:
/app/build/index.js:7667
throw new Error("Missing tiktoken_bg.wasm");
^
Error: Missing tiktoken_bg.wasm
at ../../node_modules/tiktoken/tiktoken.cjs (/app/build/index.js:7667:13)
at __require (/app/build/index.js:9:50)
at Object.<anonymous> (/app/build/index.js:58442:31)
at Module._compile (node:internal/modules/cjs/loader:1112:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1166:10)
at Module.load (node:internal/modules/cjs/loader:988:32)
at Module._load (node:internal/modules/cjs/loader:834:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
at node:internal/main/run_main_module:17:47
This looks to be caused by the postprocess script for building, which hardcodes the package path.
tiktoken/wasm/scripts/postprocess.ts
Line 177 in 71db434
If I remove the @dqdb
folder from the join arguments, I am able to run the application.
dqbd commented
Should be fixed with previous version, closing for now! Let me know if the issue persists.