equinix-labs/otel-cli

Can't create singly nested span

ham1 opened this issue · 2 comments

ham1 commented

We are using a background span like so:

export OTEL_SOCK_DIR="$(mktemp -d)"
export OTEL_JOB_CARRIER="$(mktemp)" # traceparent propagation via tempfile
# start background span for whole job
otel-cli span background \
  --tp-carrier "${OTEL_JOB_CARRIER}" \
  --sockdir "${OTEL_SOCK_DIR}" \
  --tp-print \
  --service "Build-Events" \
  --timeout 3600 \
  --name "${CI_JOB_NAME}${CI_NODE_INDEX:-}" &

then during the CI job we are creating spans:

otel-cli span -s 'Build-Events' \
  -n "${PREV_SECTION}" \
  --start ${PREV_TIME} \
  --tp-print \
  --tp-carrier "${OTEL_JOB_CARRIER}"

However when we create multiple spans like this, they seem to be continually nested, rather than all direct children of the background span.

image

Why does a non-background span update the tp-carrier, or at least not revert it?
Is this a bug or am I doing something wrong?

ham1 commented

As a workaround, I've added the following after the initial background span

export $(tail -n1 $OTEL_JOB_CARRIER)

then removed the --tp-carrier from the child spans.

Oh! Good catch. Yes, by default --tp-carrier will overwrite the previous traceparent with the current one.

I'll think about this a little bit, and either change the behavior or add a --tp-carrier-readonly option (or something like that).

Another workaround in the mean time is to use --tp-export with --tp-carrier and source the carrier file to set TRACEPARENT envvar.

Something like:

tpfile=$(mktemp)
otel-cli span background --tp-carrier $tpfile --tp-export
source $tpfile
otel-cli span # don't put --tp-carrier on this one