How to I chain traces over more than two processes?
reselbob opened this issue · 13 comments
Hi:
I am trying to chain a trace over more than two processes.
I have three services: Service_A, Service_B and Service_C.
I understand that to chain from Service_A to Service_B I do the following in psuedo-code
Service_A
parentSpanContext = tracer.extract(FORMAT_HTTP_HEADERS, request.headers)
then I do an injection in Service_A to get it to Service_B:
const span = tracer.startSpan('call_service', { childOf: root_span.context() });
tracer.inject(span, FORMAT_HTTP_HEADERS, {});
This works just fine. However, chaining the trace for Service_B to Service_C is a problem for me.
In Service_B I get the parentSpanContext, like so
parentSpanContext = tracer.extract(FORMAT_HTTP_HEADERS, request.headers)
And to forward the chain onto Service_C, I think I do the following injection in Service_B:
tracer.inject(span, FORMAT_HTTP_HEADERS, request.headers);
Where I think I should add the request.headers of Service_B to inject() to keep the chain going.
Yet, when I look at the trace in Jaeger, the spans in Service_A have an association with Service_B. Yet, the Service_B spans do not have have an association with Service_C. However, the Service_C spans do appear distinctly.
Here is the actual code for Service_B. It's the burgerqueen service.
https://github.com/reselbob/dockerdemos/blob/add_tracing_to_foodcourt/foodcourt/burgerqueen/index.js
I must be doing something very wrong. Any help will be greatly appreciated.
From the code you linked, you're making a downstream call using await axios.post, which has no relation to the headers you previously populate. Also, you're using headers from the original request to the service, but you should be creating a new request for the downstream call and injecting trace context into the new request's headers.
Thank you, let me try. BTW: I just viewed you on YouTube here: https://youtu.be/fjYAU3jayVo It's very inspiring work!
injecting trace context
Do you mean span.context() which I see here: https://opentracing-javascript.surge.sh/classes/span.html#context
Also, if I may ask a favor, do you have an example of which attribute I should use to attach the context to the request.headers?
Is this the correct way as I show with header below. Or should the new request header simply be {}?
const callPaymentService = async (payload, root_span, request) => {
const service = 'payments';
const url = `http://${service}:${port}`;
const span = tracer.startSpan('call_service', { childOf: root_span.context() });
const headers = {
context: root_span.context()
};
tracer.inject(span, FORMAT_HTTP_HEADERS, headers);
await axios.post(url, payload, {headers})
.then(res => {
span.setTag(Tags.HTTP_STATUS_CODE, 200)
span.setTag('service_call_result', res.data)
span.finish();
return res.data;
}, e => {
span.setTag(Tags.ERROR, true)
span.log({
'event': 'error',
'error.object': e
});
span.finish();
});
};
Not to wear out my welcome, @yurishkuro, but I still can't get the trace to propagate:
Here is the client code here: https://github.com/reselbob/dockerdemos/blob/add_tracing_to_foodcourt/foodcourt/burgerqueen/index.js
const callPaymentService = async (payload, root_span, request) => {
const service = 'payments';
const url = `http://${service}:${port}`;
const span = tracer.startSpan('call_service', { childOf: root_span.context() });
span.setTag(Tags.SPAN_KIND, Tags.SPAN_KIND_RPC_CLIENT)
tracer.inject(span.context(), FORMAT_HTTP_HEADERS, {});
await axios.post(url, payload, {})
.then(res => {
span.setTag(Tags.HTTP_STATUS_CODE, 200)
span.setTag('service_call_result', res.data)
span.finish();
return res.data;
}, e => {
span.setTag(Tags.ERROR, true)
span.log({
'event': 'error',
'error.object': e
});
span.finish();
});
};
And here is the receiving code here: https://github.com/reselbob/dockerdemos/blob/add_tracing_to_foodcourt/foodcourt/payments/index.js
const server = http.createServer((request, response) => {
parentSpanContext = tracer.extract(FORMAT_HTTP_HEADERS, request.headers)
const span = tracer.startSpan('payments_service_request', {
childOf: parentSpanContext,
tags: { [Tags.SPAN_KIND]: Tags.SPAN_KIND_RPC_SERVER }
});
let body = '';
if (request.method.toUpperCase() === 'POST') {
request.on('data', chunk => {
body += chunk.toString(); // convert Buffer to string
});
request.on('end', () => {
const payment = JSON.parse(body);
payment .status = 'PAID';
payment.transactionId = uuidv4();
span.setTag('payment', payment);
span.log({
'event': 'payment_service_request',
'value': payment
});
const str = JSON.stringify(payment);
span.setTag(Tags.HTTP_STATUS_CODE, 200)
span.setTag('payment_call_result', str)
response.setHeader("Content-Type", "application/json");
response.writeHead(200);
response.end(str);
span.finish();
});
}
}).listen(port, (err) => {
console.log(`${service} API Server is started on ${port} at ${new Date()} with pid ${process.pid}`);
});
Again, the spans show up in separate traces, but I can get the client to bind to the receiver is the code as shown about.
My sincerest thanks for any help you can provide.
When you call tracer.inject(span.context(), FORMAT_HTTP_HEADERS, something);, that something is better be part of the request you are making to another service. You've shown examples where something is request.headers but not used in the subsequent POST, or in the last comment something=={}, i.e. completely throw-away. Inject call populates something with trace metadata that must be received by the callee.
Did you try following https://github.com/yurishkuro/opentracing-tutorial ?
Yes, I read the Node.js tutorial, lessons 3 and 4. That calls across to two processes, publisher and formatted. This example goes one level deep, as evident by a header value of {} in http_get(). I've gotten a one-level client to server call working.
const method = 'GET';
const headers = {};
span.setTag(Tags.HTTP_URL, url);
span.setTag(Tags.HTTP_METHOD, method);
span.setTag(Tags.SPAN_KIND, Tags.SPAN_KIND_RPC_CLIENT);
// Send span context via request headers (parent id etc.)
tracer.inject(span, FORMAT_HTTP_HEADERS, headers);
Where I am getting confused is how to populate the trace metadata into the subsequent call to the second server. I am unclear as to exactly what data goes into the header value of the call to the second server in my scenario. (I am trying to hop over three microservices is sequence.) Obviously, the empty JSON object is wrong in the example below.
const header = {};
await axios.post(url, payload, {});
As you point out, it should be something. I am just unclear as to what exactly that something looks like. Might you have an example?
I understand that my confusion and questions may be a bit annoying to someone as expert as yourself. But, I trust you understand I am trying really hard to understand and am very grateful for the help you've provided. If I've exhausted your patience, I understand completely.
Thank you sincerely.
This is fine:
const headers = {};
tracer.inject(span, FORMAT_HTTP_HEADERS, headers);
but then when you make a POST request, you need to use these headers. I am not familiar with axios to advise how that can be done; usually all HTTP frameworks allow you to set the headers for outgoing requests.
Yes, the issue is some foolishness going on in how I am setting the Axios call. I will post the solution in the morning. Thank you for your patience.
Thank you, @yurishkuro ! I wish there was a way I could contact you to send you a token of appreciation for your guidance and patience.
hehe, you can https://www.buymeacoffee.com/yurishkuro
Just bought you some more coffee. I need help with a similar problem using OpenTelemetry. Please, let me know if you need more coffee than I sent you. @yurishkuro .
Have you tried their chat https://gitter.im/open-telemetry/opentelemetry-node ? I can try to help, but I don't work with OTEL libraries.
Yes, I am there now. Not getting a response. Let me go over to the OpenTracing Node Library on GitHub and open an issue. There is something going on with how I configuring docker-compose.yml. As I said things work just fine when running on bare localhost.