FoundationDB/fdb-document-layer

Assign UUID to each incoming message

Opened this issue · 2 comments

Right now it's hard to debug a specific bug that's reproducible by a query on a running production server since it's hard to locate the log (trace event log) that is related to this specific message. Thus if DocLayer can assign a UUID for each incoming message and somehow attach that UUID with all loggings produced when processing it, future debugging will be much easier.

apkar commented

Yeah, this would be super helpful.

We should probably have some kind of request context which we pass around across all functions. For example, Plan::execute() functions are unaware of the user request they are working on. We could imagine a better version of ExtMsg as the request context. Although the way current code is structured, it makes sense for it to be part of DocTransaction, as almost all functions have access to it. Considering the plan to reduce the dependency on DocTransaction we have to see what is the better fit here.

Another aspect of this problem is tracing, TraceEvent doesn't work like Java log4j framework, we can't change the log/trace level on the fly. All trace events are logged all the time. We probably don't want to enable request level tracing all the time. That would roll over logs very quickly. We should come up with a scheme to have request level logs disabled by default. We should be able to enable them with a custom MongoDB command.

Yes. This also means that once we have such UUID ready, we need to attach that UUID to the reply message we sent to the user too.