Database Spans Tagged Wrong
dwinrick-lever opened this issue · 11 comments
Expected Behavior
I expect that when using datadog-lambda-js with mongodb, the spans will be tagged with service=mongodb so that they show up in APM, including the service overview for mongodb. This is the behavior with dd-trace-js.
Actual Behavior
The spans are of type = db, but nothing shows in the APM dashboard for the mongodb service overview. I do see traces, but they are categorized wrong.
I'm hoping this is a configuration issue somewhere as that would be easy to fix.
A trace being sent to the forwarder:
We can see that, instead of the normal differentiation between the service and the database spans, DD does know that it's for the database, but the spans are the same colors as everything else.
List of queries that should show "mongodb" in the first column:
APM dashboard for mongodb is empty:
Some of the tags of one of the spans in question, from the UI.
Specifications
- Datadog Lambda Layer version: ^4.67.0
- Node version: 14.x
Hmm, actually this is more like a product question, as this is indeed the expected behavior -- all the spans under a Lambda invocation inherit the same service name. Do you mind opening a support ticket or contact your Datadog CSM to set up a quick call with our product team to better understand the intention and problems?
@tianchu Thanks for the quick reply. Let me explain why this does not seem like a product question.
I expect that product would say that APM should work with lambda, especially considering DD's marketing.
The reason this seems like a bug is that you can still attach metadata saying it's from the lambda, just like dd-trace does, in which case you can still see which services the query is from. For example, when using dd-trace with Node, I see that queries are correctly classified, but I am still able to see what service they are from.
Expected query list, as with all other integrations:
We can still see which service the query is from:
I would expect datadog-lambda-js would behave the same. In fact, if you use AWS services, this library does tag service=dynamodb, or whatever, but not for Mongo...
Let me know if you still want me to contact my CSM, but as of right now this is blocking a project that needs this data to move forward with making production changes...
Looking into this more, not sure it's related to the service tag.
Hooking into dd-trace/src/encode, and comparing the data sent to DD in lambda vs in our services, I see that two paths are missing when using lambda:
meta.env
meta['mongodb.query']
Going to do some testing and see if this is actually the problem.
Figured it out! Monkey-patching dd-trace/src/encode to set meta.env and meta['mongodb.query'] fixes the APM service page!
Next - figuring out exactly which one of these is the issue, if not both, and why they are not populated...
@dwinrick-lever Sorry for the inconvenience and really glad to hear that you can get things to work in the way you liked. Could it be related to the version of dd-trace-js used in your lambda service and the other service you mentioned?
@tianchu We're using latest of everything, do you have other recommendations?
I would hardly say patching an internal file in an external library is the preferred approach :)
We didn't find the bug yet, it's either in dd-trace or dd-lambda-js, for some reason some meta is getting stripped. How would you like to move forward to get this fixed?
This is our current workaround, which is a bit more high level than the encoding libraries:
const originalStartSpan = dataDogTracer.prototype.startSpan;
dataDogTracer.prototype.startSpan = function (name: string, options: {}) {
const span = originalStartSpan.call(this, name, options);
if (span._spanContext && span._spanContext._tags) {
if (!span._spanContext._tags.env) {
span._spanContext._tags.env = process.env.LEVER_ENV;
}
if (span._spanContext._tags['span.type'] === 'mongodb' && !span._spanContext._tags['mongodb.query']) {
span._spanContext._tags['mongodb.query'] = span.resource.substring(span.resource.indexOf('{'), span.resource.length);
}
}
return span;
}
It is still not known why this is needed for lambda.
Hmm, that's interesting. The missing env seems to be a known issue (the trace should still have the correct env tagged, it just doesn't show up in the UI). The missing mongodb.query is probably related to this if, but I'm not the expert of that code base (dd-trace-js).
We will triage this ticket internally and assign an engineer to work on this. First we'll see if we can reproduce the issue, if yes, the fix is probably not hard.
Hey @tianchu any luck here?