Kong/kong-plugin-zipkin

unknown-service-name in Jaeger when localEndpoint.serviceName is missing

goober opened this issue · 14 comments

goober commented

We are using Jaeger for collecting trace-information from our services and have just installed the zipkin-plugin for Kong (v1.0.3). However, the serviceName for kong is missing, and instead unknown-service-name is presented, as seen in the screenshot below.

image

As of my understanding, Jeager takes the serviceName from the localEndpoint, which works fine for the kong.proxy span that can be seen in the screenshot above and in reporter.lua#L42-L54.
#12 resolved the above problem in cases where the the peer.service exists, but I expected that a default value for the serviceName would be set to kong in other cases.

Maybe this is a duplicate of #23 and if that is the case we could continue the discussion in that issue?

Update
This is how Jaeger determines the serviceName from a zipkin span:
to_domain.go#L206-L233

james-callahan commented

What would you like to see for each of those spans instead?

goober commented

That is a great question, I hoped that you would have some answers to that :)

But as a minimum it would be great to at least default to kong or if it is configurable though an environment-variable as suggested in #23.

For spans that are children of auth-service in this case I would have expected that they would have the same serviceName as their parent. i.e. auth-service

  • 👍4
james-callahan commented

That is a great question, I hoped that you would have some answers to that :)

I have several answers, none of which I'm totally happy with 😛

One problem I think about is how kong is "pretending" to be the service. However from the POV of the client: it is the service. But from the POV of the service, kong is just a client.

It can get a bit complex with how things might evolve: e.g. if within the access phase, we reach out to ldap, then there should be a span related to that ldap query: should the localEndpoint serviceName still be "auth-service" for that case (and I the remoteEndpoint serviceName would be my-ldap-server or similar)

goober commented

I agree with you that it could get complex in some cases.

Is it too early in the chain to get the consumer.id in the kong.request span? Otherwise could that be a candidate for the service name at the top of the chain until the proxied service name is available?

I would argue that it can be of interest to see the chain from the service point of view, including kong as a client to your service, i.e. setting the serviceName to kong. But from your ldap example it is not a clear answer for how I would expect it to work.

james-callahan commented

Is it too early in the chain to get the consumer.id in the kong.request span?

Theoretically, yes it's too early. In practice, sometimes it's possible. Despite the span starting before we know where the request is going, we don't currently ship the log to zipkin until after the request has finished, which means we can edit it before it is sent. However not all requests will have a (known) service; and also in the future to support long running requests, we're considering sending the logs before the log phase.

As a jaegar/zipkin user: what do you expect to see for in the service field(s) for those spans? e.g. should some contain "my-kong-proxy"?

  • 👍1
goober commented

I would have expected the first two spans to be "my-kong-proxy", i.e. the servicename for the kong instance. For the spans which have auth-service as parent I would be totally fine with either "my-kong-proxy" or the service name (auth-service)

james-callahan commented

I would have expected the first two spans to be "my-kong-proxy", i.e. the servicename for the kong instance.

Where does that string "my-kong-proxy" exist in your infrastructure? Should that be a plugin config option? Grab it from the nginx server_name variable? (though that's almost always just "kong", so not so useful)

goober commented

We are running kong in a kubernetes environment. So from our point of view I would have preferred an environment variable what we can specify from the outside.
Then we can map it in our Deployment as:

env:
 - name: KONG_ZIPKIN_PLUGIN_SERVICE_NAME
     valueFrom:
       fieldRef:
         fieldPath: metadata.name

This is also suggested in #23

  • 👍3
jeremyjpj0916 commented

Just saw this as I was hitting those unknown spans too, would it make sense to start the first one to populate with kong or a config value. Then once kong knows the service name to allow that tag to be the service name through the rest of the proxy phases? Plenty of possibilities though.

Edit -

Looking into the codebase a bit I decided to log to stdout the json that gets sent:


2019/04/01   17:19:27 [error] 36#0: *4094 [lua] reporter.lua:103: flush():   [{"traceId":"1edb83ef82c3b3a0fa90ef5e46010ee5","localEndpoint":null,"id":"c6ce85f466723f59","tags":{},"name":"kong.header_filter","parentId":"edcb4c24a3321e5c","remoteEndpoint":null,"duration":0,"timestamp":1554139167979000},{"traceId":"1edb83ef82c3b3a0fa90ef5e46010ee5","localEndpoint":null,"id":"6729bc2a8c84447a","tags":{},"name":"kong.rewrite","parentId":"9302a8aa206d54d5","remoteEndpoint":null,"duration":0,"timestamp":1554139167979000},{"traceId":"1edb83ef82c3b3a0fa90ef5e46010ee5","localEndpoint":null,"id":"7667198b340a91b6","tags":{},"name":"kong.access","parentId":"edcb4c24a3321e5c","remoteEndpoint":null,"duration":3000,"timestamp":1554139167979000},{"traceId":"1edb83ef82c3b3a0fa90ef5e46010ee5","localEndpoint":null,"id":"f416a4956ef91818","tags":{},"name":"kong.body_filter","parentId":"edcb4c24a3321e5c","remoteEndpoint":null,"duration":3000,"timestamp":1554139167979000},{"traceId":"1edb83ef82c3b3a0fa90ef5e46010ee5","localEndpoint":{"serviceName":"F5MonitorService"},"id":"edcb4c24a3321e5c","tags":{"kong.route":"dcc39b07-2723-4d4d-8e89-e8a92301b47d","kong.service":"11286847-6046-495a-be17-70a42f3b8c3b"},"name":"kong.proxy","parentId":"9302a8aa206d54d5","kind":"CLIENT","remoteEndpoint":{"port":8001},"duration":3000,"timestamp":1554139167979000},{"traceId":"1edb83ef82c3b3a0fa90ef5e46010ee5","localEndpoint":null,"id":"9302a8aa206d54d5","tags":{"kong.node.id":"dce51e78-b1c1-40c4-a90c-f56a62e23899","http.method":"GET","http.status_code":"200","http.url":"https:\/\/gateway-dev-core.company.com:8443\/Health\/check","component":"kong"},"name":"kong.request","kind":"SERVER","remoteEndpoint":{"ipv4":"10.131.48.1","port":58620},"duration":3000,"timestamp":1554139167979000}],   context: ngx.timer, client: 10.131.48.1, server: 0.0.0.0:8443

@james-callahan What do you think about just having the first span localEndpoint say Some configurable value from the schema, defaults to Kong, or null as it behaves now but could be renamed to whatever a team wants? Then rather than just carrying the "localEndpoint":{"serviceName":"F5MonitorService"} at the kong.proxy phase only why not set it at every phase other than the first "configed" one? I could probably get my group to PR something like that if yall see it as appropriate, any impl is better than unknown-service-name in almost all spans haha.

  • 👍1
palmerabollo commented

From a jaeger-collector's point of view, logs are full of warning messages due to this issue:

{"level":"warn","ts":1559690330.4296474,"caller":"app/thrift_span_handler.go:136","msg":"Warning while converting zipkin to domain span","error":"cannot find service name in Zipkin span [traceID=f466d9100ea16133, spanID=92f495f6206fcb89]"}

james-callahan commented

We are running kong in a kubernetes environment. So from our point of view I would have preferred an environment variable what we can specify from the outside.

Wouldn't you want Kong to have different service names depending on which service it is proxying for/to?

secat commented

IMHO I think it depends if Kong is proxying north/south traffic or if Kong is proxying east/west traffic.

In the first case (i.e. Kong proxying traffic north/south), the suggestion proposed by @goober would be a good solution.

goober commented

@secat our current setup is a north-south type where only ingress requests are proxied through kong. All in-cluster communication between services are handled directly without going through kong. That is why we think it would solve our issue if the kong instance could be named separately to the service that was actually called.

  • 👍1
secat commented

@james-callahan we are currently deploying three (3) kong gateway on our kubernetes cluster and we would like to discriminate from which Kong proxy the traffic is coming from.

Having the possibility to configure the service name as poposed in @goober comment would be much appreciated.

  • 👍3