After adding mapping template no docs indexed
Rassi opened this issue · 1 comments
I attempted to fix mapping conflicts by adding a mapping template, but after doing that no logs were added to new indexes.
I got no exceptions, and the internal nlog log file didn't show anything. After getting the NLog.Targets.ElasticSearch
code, I could finally set a breakpoint in ElasticSearchTarget.SendBatch
and saw the following at https://github.com/markmcdowell/NLog.Targets.ElasticSearch/blob/master/src/NLog.Targets.ElasticSearch/ElasticSearchTarget.cs#L290.
result.DebugInformation
:
Successful (200) low level call on POST: /_bulk
# Server indicated deprecations:
- 299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Specifying types in bulk requests is deprecated."
# Audit trail of this API call:
- [1] PingSuccess: Node: https://www.example.com:9243/ Took: 00:00:00.2835193
- [2] HealthyResponse: Node: https://www.example.com:9243/ Took: 00:00:00.4522220
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
# Response:
{"took":2,"errors":true,"items":[{"index":{"_index":"testindex-03","_type":"logevent","_id":"JVDaA3IBc9a3A5n7JPnq","status":400,"error":{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [testindex-03] as the final mapping would have more than 1 type: [_doc, logevent]"}}}]}
So it is caused by Elastic deprecating types, and me having specified documentType as logevent (don't ask me why. Must be copy/pasta from somewhere.)
nlog.config:
<nlog xmlns="http://nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
throwExceptions="true"
internalLogFile="c:\temp\nlog_internal_log_file.txt">
<extensions>
<add assembly="NLog.Targets.ElasticSearch"/>
</extensions>
<targets async="true">
<target xsi:type="ElasticSearch"
name="elastic"
uri="${configsetting:name=Logging.ElasticUri}"
index="testindex-03"
documentType="logevent"
includeAllProperties="True"
requireAuth="True"
username="${configsetting:name=Logging.ElasticUser}"
password="${configsetting:name=Logging.ElasticPassword}">
<field name="app" layoutType="System.Object">
<layout xsi:type="JsonLayout" IncludeGdc="true" />
</field>
<field name="logger" layout="${logger}" />
<field name="request" layoutType="System.Object">
<layout xsi:type="JsonLayout" >
<attribute name="client-ip" layout="${mdlc:client-ip}" />
<attribute name="client-id" layout="${mdlc:client-id}" />
<attribute name="client-name" layout="${mdlc:client-name}" />
<attribute name="duration-ms" layout="${mdlc:duration-ms}" encode="false" />
<attribute name="status-code" layout="${mdlc:status-code}" encode="false" />
<attribute name="url" layout="${mdlc:url}" />
<attribute name="request-id" layout="${mdlc:RequestId}" />
<attribute name="request-path" layout="${mdlc:RequestPath}" />
</layout>
</field>
</target>
</targets>
<rules>
<logger name="*" minLevel="Trace" appendTo="elastic" />
</rules>
</nlog>
So document type logevent
conflicts with the default _doc
when index/mapping is created from a template. I can see that logevent
is default in https://github.com/markmcdowell/NLog.Targets.ElasticSearch/blob/master/src/NLog.Targets.ElasticSearch/ElasticSearchTarget.cs#L149, and when I tried removing documentType
from my nlog.config
it did indeed still fail with the same error.
So I fixed it by changing to the same document type (_doc
), but I would still like to point out a couple of suggestions:
- Default document type to
null
or_doc
(seems like others have set it to null https://issues.apache.org/jira/browse/KARAF-6539 and worked for me when I setElasticSearchTarget.DocumentType
to null runtime) - Raise exception or log internal error when receiving these kinds of "nested" errors.