review semantic conventions and Elastic Common Schema

[loomis]

Review the semantic conventions and Elastic Common Schema to determine if there are changes necessary in the generate attribute names to improve compatibility.

[loomis]

Overview

The OpenTelemetry semantic conventions for logs are experimental meaning that breaking changes are allowed in the specification.

The Elastic Common Schema (ECS) is stable. Version 8.7 is the latest revision.

Note that the ECS has been donated to the OpenTelemetry consortium and that there will be a merger between the ECS and the OpenTelemetry semantic conventions. See the OpenTelemetry announcement and the Elastic announcement.

#Comparison

The following table provides links to the sections of the two specifications that roughly cover the same areas.

Semantic Conventions	Elastic Common Schema
index	index
general	base fields
events	event fields
exceptions	error fields
feature flag
log media	log fields

General

The general section of the OpenTelemetry specification only defines log.record.uid. This is intended to be used for deduplication use cases. The Serilog OpenTelemetry sink does not deal with deduplication. For Serilog, this seems more appropriate for a separate processor/filter.

IMO, this should not have any impact on the Serilog OpenTelemetry Sink v1.0.0 release.

Events

The definition of events in the two schemas are significantly different. However, this deals with the detailed content of the log record, created by the developer. Developers can decide which specification to follow (depending on context) and emit the correct attributes.

These attributes do not have an impact on the Serilog OpenTelemetry Sink v1.0.0 release.

Exceptions

The semantic conventions and the ECS define essentially the same attributes for exceptions, but with different prefixes. ECS uses the error prefix and the semantic conventions use the exception prefix. Currently, the Serilog OpenTelemetry Sink follows the OpenTelemetry semantic conventions.

Questions:

• Should the sink switch to using the ECS attributes?
• Should the sink provide an option to choose between the ECS and semantic conventions?

Feature Flag

AFAICT, there is no comparable set of attributes/conventions within the ECS. Like for events, developers can choose which convention to follow. The sink just transmits the attributes that have been chosen.

These attributes do not have an impact on the Serilog OpenTelemetry Sink v1.0.0 release.

Log Media

Generally, the ECS provides a more complete set of attributes. Where they do overlap, the names differ. However, the Sink itself does not deal with log source information; that is a function of Serilog as a whole. While Serilog core may want to make a choice here (or provide options), those choices do not affect the implementation of the Serilog OpenTelemetry Sink.

These attributes do not have an impact on the Serilog OpenTelemetry Sink v1.0.0 release.

Summary

See the questions concerning the representation of exceptions. The choices are to support ECS, support the semantic conventions, or support both with an option.

Feedback on the comparison and choice is welcome.

[loomis]

Although the semantic conventions and ECS do not define anything similar to a message template, the ECS does define conventions around hashes. Given the convergence between the two specs, it would probably be prudent to rename message_template.md5_hash to message_template.hash.md5, which follows the ECS pattern.

[loomis]

With more thought, I think that the implementation should stick to following the semantic conventions for the exception information. We don't yet know what the convergence of the specs will look like, there is vendor tooling in place that follows the semantic conventions, and these attributes can be trivially renamed in any log processor, if needed.

[loomis]

Summary of things to be done:

• No-op on the exception attributes; leave the current naming in place.
• Rename message_template.md5_hash to message_template.hash.md5
• Fix README doc on flag for enabling message template hash

Unless there are objections to the choices above, I'll implement those.