Can't log JSON to ElasticSearch and have it treated as type=object

[BuddhaBuddy1]

I am trying to have my payload.message treated as an object, but it always indexes as text. I have configured my "outputs" to successfully log text to ElasticSearch, but I want to have it as JSON instead. Everything I find says that it needs to be configured in mappings.properties, but I can't find any Diagnostics Eventflow specific examples. I tried using the example from the Readme.MD and just changing it from timestamp to "message", and the type to "object". I started my application, and then deleted all indexes and even the index mapping. I then triggered a log, but it still gets mapped as "text". I then tried putting "message" under "payload", but that didn't work either (after deleting index and index mapping again).

      "mappings": {
        "properties": {
          "payload": {
            "message": {
              "type": "object"
            }
          }
        }
      }

Is it possible to log an object to ElasticSearch with this library? I don't see object listed in Microsoft.Diagnostics.EventFlow.Outputs.CreateIndexRequestBuilder.typeToPropertiesDesctiptorFunc, which seems like a bad sign. However, I also don't see any health warnings, and the message actually appears, it is just indexed as text, so I can't search a particular field in the JSON.

[karolz-ms]

@BuddhaBuddy1 the answer is yes, but it depends on the EventFlow input you are using, and the Elasticsearch version you are using. Here is an example that works with Elasticsearch 7.2 and Microsoft.Extensions.Logging (.NET Core):

using System;

using Microsoft.Diagnostics.EventFlow.Inputs;
using Microsoft.Diagnostics.EventFlow.HealthReporters;
using Microsoft.Diagnostics.EventFlow.Outputs;
using Microsoft.Diagnostics.EventFlow.Configuration;
using Microsoft.Diagnostics.EventFlow;

using Microsoft.Extensions.Logging;
using System.Collections.Generic;

namespace EventFlowLoggerToES
{
    class Program
    {
        static void Main(string[] args)
        {
            CsvHealthReporter healthReporter = new CsvHealthReporter(new CsvHealthReporterConfiguration());
            LoggerInput loggerInput = new LoggerInput(healthReporter);
            StdOutput stdOutput = new StdOutput(healthReporter);

            ElasticSearchOutputConfiguration esConfiguration = new ElasticSearchOutputConfiguration();
            esConfiguration.IndexNamePrefix = "objectmappingtest-";
            esConfiguration.ServiceUri = "http://localhost:9200";
            ElasticSearchOutput elasticSearchOutput = new ElasticSearchOutput(esConfiguration, healthReporter);

            var pipeline = new DiagnosticPipeline(
                healthReporter,
                new[] { loggerInput },
                globalFilters: null,
                new[] { new EventSink(stdOutput, null), new EventSink(elasticSearchOutput, null) },
                pipelineConfiguration: null,
                disposeDependencies: true
            );

            using (pipeline) {
                var loggerFactory = new LoggerFactory();
                loggerFactory.AddEventFlow(pipeline);
                var logger = new Logger<Program>(loggerFactory);

                logger.LogInformation("Informational message {number} with data {data}", 
                    (new Random()).Next(),
                    new Dictionary<string, int>
                    {
                        ["alpha"] = 1,
                        ["bravo"] = 2
                    }
                );
            }
        }
    }
}

result:

    {
        "_index": "objectmappingtest--2019.12.14",
        "_type": "_doc",
        "_id": "sp-hAm8BmKNnr6KoGi8R",
        "_score": 1,
        "_source": {
          "timestamp": "2019-12-14T04:18:29.2772393+00:00",
          "providerName": "EventFlowLoggerToES.Program",
          "level": 4,
          "keywords": 0,
          "payload": {
            "Message": "Informational message 1318857574 with data [alpha, 1], [bravo, 2]",
            "EventId": 0,
            "number": 1318857574,
            "data": {
              "alpha": 1,
              "bravo": 2
            }
          }
        }
      }

Also note that type mappings are deprecated in ES 6 and removed in ES 7

Does this help?

[BuddhaBuddy1]

I'm using Azure Service Fabric, which uses the EventSource input. That might be where the limitation is. I was messing with type mappings because that seemed the only way to try to change it from being treated as a string.
I might have to try using a second input, and something other then EventSource for my own logging. Other limitations of EventSource were driving me crazy, anyway. :)
Just to be clear, there was nothing special in your example configuration that made it possible to log an object, right? You just passed an object in your logging method, and it worked as expected?

[karolz-ms]

I'm using Azure Service Fabric, which uses the EventSource input. That might be where the limitation is.

That is right, properties of EventSource events can only be simple types like numbers, strings, or GUIDs.

Just to be clear, there was nothing special in your example configuration that made it possible to log an object, right? You just passed an object in your logging method, and it worked as expected?

Correct. The Microsoft.Extensions.Logging input leverages the capability of .NET Core loggers to pass objects around and pushes them through the EventFlow pipeline to EventFlow Elasticsearch output. In turn, the output passes them unchanged (as event payload properties) to NEST BulkIndexOperation. The logic inside BulkIndexOperation then kicks in and serializes objects as Javascript objects.

Also note that in the example above I have set up EventFlow entirely through code. The example does not use any configuration.

[BuddhaBuddy1]

In your example, it looks like it has to be part of the message to be logged. I would like to log an object that is NOT part of the message, so it would be more like:

    {
        "_index": "objectmappingtest--2019.12.14",
        "_type": "_doc",
        "_id": "sp-hAm8BmKNnr6KoGi8R",
        "_score": 1,
        "_source": {
          "timestamp": "2019-12-14T04:18:29.2772393+00:00",
          "providerName": "EventFlowLoggerToES.Program",
          "level": 4,
          "keywords": 0,
          "payload": {
            "Message": "Informational message 1318857574",
            "EventId": 0,
            "number": 1318857574,
            "data": {
              "alpha": 1,
              "bravo": 2
            }
          }
        }
      }

That doesn't seem possible with ILogger, as all the methods are limited to a formatted string with parameters. Can this only be done with something like a Serilog Enricher?

[karolz-ms]

Yes, I found this lengthy discussion in Microsoft Extensions repo; seems to be exactly about what you are trying to do with ILogger.

I am not sure about Serilog enrichers, but there are two alternatives that most certainly will work. One is to use DiagnosticSource that has its own EventFlow input. Alternatively you can write your custom EventFlow input, which is just a class that implements IObservable<EventData>. I would try DiagnosticSource first; this is the logging API that is used by many .NET Framework and ASP.NET libraries, and it is similar to EventSource, so the code change will be small if you are already using EventSource.

[BuddhaBuddy1]

Thanks, that thread is exactly what I'm trying to do! I tried with a Serilog input, but still using MS.Diagnostics.EventFlow.Output.ElasticSearch, but I must be doing something basically wrong because my events are not getting logged at all. At least now I have several options.