Module not found: Can't resolve 'tiktoken'

[Tarektouati]

I just upgraded to v5.17.0 from v5.10.0

since i'm getting this error logs in my console :

⚠ ../../node_modules/dd-trace/packages/datadog-plugin-openai/src/index.js
Module not found: Can't resolve 'tiktoken' in 'node_modules/dd-trace/packages/datadog-plugin-openai/src'

I don't use any OpenAPI or LLM stuff,

Next.js application deployed on GKE.
Node version : v20.12.1
Next.JS version : 14.2.2

Can you help on this please ?

[wlechowicz]

+1

v5.12 added OpenAI support so presumably from that version up this error shows up.

I installed dd-trace for the first time and it gave me v5.17, I used dd-trace/initialize to instrument my app, which seems to have been added in 5.15 (not mentioned in release notes). I tried 5.15 and 5.16 and same tiktoken error. 5.14 doesn't have the initialize script that I need so no reason to try lower versions.

Seems like the plugin manager is badly designed, why is it loading plugins that aren't used? If I was using this plugin, maybe I wouldn't mind installing tiktoken myself, as this looks like a widespread issue. Instead ,it produces an error for a plugin that's not used even by a bare-bones implementation with just init.

[sabrenner]

Hi @Tarektouati @wlechowicz! Thanks for flagging this issue. We added support for capturing token metric specifically for the OpenAI integration in 5.15.0. We also added some safeguards to handle the case that there is no tiktoken module installed, so I'm intrigued as to why this error is showing up.

I tried reproducing this locally, but was unable to. Are you doing anything other than running node --import dd-trace/initialize.mjs index.js (or whatever your entrypoint file is)? If you're able to provide a small snippet or sample that led to this error, it would be awesome to have on my end, thanks!

In the meantime, you can try setting DD_TRACE_DISABLED_PLUGINS=openai and/or DD_TRACE_OPENAI_ENABLED=false to see if that helps this issue at all, until we get a proper solution in place.

And, an additional question for @wlechowicz , out of curiosity, are you also using Next.js?

[sabrenner]

Actually, on second though, this is likely a webpack issue, whether it be under the hood of Next.js, some other framework using webpack, or just using webpack itself. Try adding "tiktoken": false to resolve.fallback, either in webpack.config.js itself, or to your nextConfig in next.config.js

webpack: (config) => {
    config.resolve.fallback = {
      tiktoken: false,
    };

    return config;
  },

Let me know if this helps!

[Tarektouati]

Hey @sabrenner 👋🏼 !

Thanks for your help on this one.

I tested the booth solutions you suggested:

In the meantime, you can try setting DD_TRACE_DISABLED_PLUGINS=openai and/or DD_TRACE_OPENAI_ENABLED=false to see if that helps this issue at all, until we get a proper solution in place.

• First one, didn't work for me,

• Second one with resolve.fallback in next.config.js worked fine, i no longer see tiktoken related error logs in my console.

I can live with this for now in codebase 😅

Is something planned on this one ?

Thanks

[wlechowicz]

@sabrenner thanks, I was indeed in the scope of a Next.js 14 app. I added the resolve.fallback as you suggested and the errors went away.

[sabrenner]

Glad to hear that workaround was useful! I've actually merged in a change that wraps this optional require in a function, so it sort of bypasses any webpack compilation, since it doesn't require tiktoken directly. I've verified this locally by just running webpack on a small script that imports dd-trace with these changes. Once it lands in the next tracer release, feel free to try and remove the workaround and try the new version out and let me know if you run into any more issues!

[prescottprue]

@sabrenner Thanks for the fix, I see #4433 was merged, but any idea when that will be released?

We see the associated warning on 20.14.0 and the proposed webpack config fixes the warning, but for Node 20.15.0 we get the following error:

TypeError: An error occurred while loading instrumentation hook: Cannot read properties of undefined (reading '0')
     at isInsideNodeModules (node:internal/util:508:17)
     at showFlaggedDeprecation (node:buffer:178:8)
     at new Buffer (node:buffer:266:3)
     at alloc (webpack-internal:///(instrument)/./node_modules/msgpack-lite/lib/bufferish-buffer.js:18:10)
     at eval (webpack-internal:///(instrument)/./node_modules/msgpack-lite/lib/bufferish-buffer.js:6:54)
...
     at (instrument)/./node_modules/dd-trace/index.js (/Users/scott/Dev/Code/reside-eng/identity-ui/.next/server/vendor-chunks/dd-trace.js:97:1)

[sabrenner]

Hi @prescottprue! We're working on cutting a new release now, it should be out sometime this week. Thanks for your patience!

For the error you attached: it doesn't immediately seem that it's related to this issue (but, correct me if you have more insight beyond that stack trace). Let's see if the next release of the tracer helps your use case, and if not, I would suggest opening a ticket with the Datadog Helpdesk to expedite the process at having someone from our team look at it.

[balamanoharb]

@sabrenner Thanks for the fix, I see #4433 was merged, but any idea when that will be released?

We see the associated warning on 20.14.0 and the proposed webpack config fixes the warning, but for Node 20.15.0 we get the following error:

TypeError: An error occurred while loading instrumentation hook: Cannot read properties of undefined (reading '0')
     at isInsideNodeModules (node:internal/util:508:17)
     at showFlaggedDeprecation (node:buffer:178:8)
     at new Buffer (node:buffer:266:3)
     at alloc (webpack-internal:///(instrument)/./node_modules/msgpack-lite/lib/bufferish-buffer.js:18:10)
     at eval (webpack-internal:///(instrument)/./node_modules/msgpack-lite/lib/bufferish-buffer.js:6:54)
...
     at (instrument)/./node_modules/dd-trace/index.js (/Users/scott/Dev/Code/reside-eng/identity-ui/.next/server/vendor-chunks/dd-trace.js:97:1)

Hi prescottprue,

I'm getting the same error, were you able to find the solution for this ?