windows_eventlog2 parsing problems
MattiDeGrauwe opened this issue · 8 comments
Hi everyone,
A while ago we started using the windows_eventlog2 for fetching server's logs on a big infrastructure.
The issue we're encountering at this point is that certain logs aren't being parsed properly to Elasticsearch (mapper_parsing_exception).
The logs that are encountering issues look like this:
{"ProviderName"=>"Microsoft-Windows-Security-Auditing",
"ProviderGUID"=>"{54849625-5478-4994-a5ba-3e3b0328c30d}",
"EventID"=>"4672",
"Level"=>"0",
"Task"=>"12548",
"Opcode"=>"0",
"Keywords"=>"0x8020000000000000",
"TimeCreated"=>"2019-10-22T16:00:01.885659600Z",
"EventRecordID"=>"328833",
"RelatedActivityID"=>"{fd2f1b03-7f96-0001-751b-2ffd967fd501}",
"ThreadID"=>"6592",
"Channel"=>"Security",
"Computer"=>"patch002.infra.be.sentia.cloud",
"Version"=>"0",
"DescriptionTitle"=>"Special privileges assigned to new logon.",
"subject.security_id"=>"S-1-5-21-2310396381-124451520-3033224903-1522",
"subject.account_name"=>"schtasks-patch002",
"subject.account_domain"=>"INFRA",
"subject.logon_id"=>"0x19296048",
"privileges"=>"SeSecurityPrivilege",
""=>"SeDelegateSessionUserImpersonatePrivilege"}
As you can see, the "privileges" field is most likely breaking the mapping.
Is there any possiblity to parse the privileges field to something else? Perhaps a comma-separated string?
Thanks in advance!
Warm regards,
Matti
@cosmo0920 do you have any idea whats going on with this field?
Could you show us your Elasticsearch cluster error log and using Elasticsearch template?
Hi Cosmo,
The elasticsearch just rejects the entry since the privileges gets split up by the plugin into multiple fields (resulting in the empty key for SeDelegateSessionUserImpersonatePrivilege).
Elasticsearch doesn't throw any error, it is td-agent that throws the folowing errors:
2019-10-23 12:08:53.339351060 +0200 fluent.warn: {"error":"#<Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError: 400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'failed to parse'>","tag":"infra.windows","time":1571825321,"record":{"ProviderName":"Microsoft-Windows-Security-Auditing","ProviderGUID":"{54849625-5478-4994-a5ba-3e3b0328c30d}","EventID":"4672","Level":"0","Task":"12548","Opcode":"0","Keywords":"0x8020000000000000","TimeCreated":"2019-10-23T10:08:40.201276000Z","EventRecordID":"330580","RelatedActivityID":"{fd2f1b03-7f96-0001-751b-2ffd967fd501}","ThreadID":"9228","Channel":"Security","Computer":"patch002.infra.be.sentia.cloud","Version":"0","DescriptionTitle":"Special privileges assigned to new logon.","subject.security_id":"S-1-5-18","subject.account_name":"SYSTEM","subject.account_domain":"NT AUTHORITY","subject.logon_id":"0x3E7","privileges":"SeAssignPrimaryTokenPrivilege","":"SeDelegateSessionUserImpersonatePrivilege"},"message":"dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error=\"400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'failed to parse'\" location=nil tag=\"infra.windows\" time=2019-10-23 12:08:41.696822000 +0200 record={\"ProviderName\"=>\"Microsoft-Windows-Security-Auditing\", \"ProviderGUID\"=>\"{54849625-5478-4994-a5ba-3e3b0328c30d}\", \"EventID\"=>\"4672\", \"Level\"=>\"0\", \"Task\"=>\"12548\", \"Opcode\"=>\"0\", \"Keywords\"=>\"0x8020000000000000\", \"TimeCreated\"=>\"2019-10-23T10:08:40.201276000Z\", \"EventRecordID\"=>\"330580\", \"RelatedActivityID\"=>\"{fd2f1b03-7f96-0001-751b-2ffd967fd501}\", \"ThreadID\"=>\"9228\", \"Channel\"=>\"Security\", \"Computer\"=>\"patch002.infra.be.sentia.cloud\", \"Version\"=>\"0\", \"DescriptionTitle\"=>\"Special privileges assigned to new logon.\", \"subject.security_id\"=>\"S-1-5-18\", \"subject.account_name\"=>\"SYSTEM\", \"subject.account_domain\"=>\"NT AUTHORITY\", \"subject.logon_id\"=>\"0x3E7\", \"privileges\"=>\"SeAssignPrimaryTokenPrivilege\", \"\"=>\"SeDelegateSessionUserImpersonatePrivilege\"}"}
There is no static template used on Elasticsearch, which means that the privileges field will get automatically mapped to a String.
Warm regards,
Matti
The elasticsearch just rejects the entry since the privileges gets split up by the plugin into multiple fields (resulting in the empty key for SeDelegateSessionUserImpersonatePrivilege).
Elasticsearch doesn't throw any error,
Really? Elasticsearch server complains the following error:
{"type": "server", "timestamp": "2019-10-28T05:04:06,657Z", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "docker-cluster", "node.name": "e8d4eeb14df6", "message": "[logstash-2019.10.28][0] failed to execute bulk item (index) index {[logstash-2019.10.28][_doc][rO2_EG4By44qHZqa95Pw], source[{\"ProviderName\":\"Microsoft-Windows-Security-Auditing\",\"ProviderGUID\":\"{54849625-5478-4994-a5ba-3e3b0328c30d}\",\"EventID\":\"4672\",\"Level\":\"0\",\"Task\":\"12548\",\"Opcode\":\"0\",\"Keywords\":\"0x8020000000000000\",\"TimeCreated\":\"2019-10-22T16:00:01.885659600Z\",\"EventRecordID\":\"328833\",\"RelatedActivityID\":\"{fd2f1b03-7f96-0001-751b-2ffd967fd501}\",\"ThreadID\":\"6592\",\"Channel\":\"Security\",\"Computer\":\"patch002.infra.be.sentia.cloud\",\"Version\":\"0\",\"DescriptionTitle\":\"Special privileges assigned to new logon.\",\"subject.security_id\":\"S-1-5-21-2310396381-124451520-3033224903-1522\",\"subject.account_name\":\"schtasks-patch002\",\"subject.account_domain\":\"INFRA\",\"subject.logon_id\":\"0x19296048\",\"privileges\":\"SeSecurityPrivilege\",\"\":\"SeDelegateSessionUserImpersonatePrivilege\",\"@timestamp\":\"2019-10-28T14:04:05.289882000+09:00\"}]}", "cluster.uuid": "wnS6ZcaHQiShKPtIaYBKTA", "node.id": "-Lm4tThgS16QL1CG-YEjXQ" ,
"stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: failed to parse",
"at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:191) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:74) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:776) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:753) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:725) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:258) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:161) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:193) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:118) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:917) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:108) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:394) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:316) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$21(IndexShard.java:2753) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:112) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2727) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:858) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:312) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:275) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) [x-pack-security-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) [x-pack-security-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:752) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.0.jar:7.4.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]",
"Caused by: java.lang.IllegalArgumentException: field name cannot be an empty string",
"at org.elasticsearch.index.mapper.DocumentParser.splitAndValidatePath(DocumentParser.java:211) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:409) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112) ~[elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) ~[elasticsearch-7.4.0.jar:7.4.0]",
"... 34 more"] }
Elasticsearch got to handle event by removing empty key record:
<filter test.*>
@type record_modifier
remove_keys _dummy_
<record>
_dummy_ ${if record.has_key?(''); record.delete(''); end; nil}
</record>
</filter>That could be an approach, but then we lose all data in the "privileges" field.
I've created a PR to fix this issue. #31
The root cause of this issue are:
- Parsed privileges record wrongly and generated empty key record.
- Empty key record collision was occurred. Then, additional privileges information were almost disappeared except for the last one.
Thank you cosmo!
How will I make use of the updated plugin? (since this is included in fluentd's core).
Warm regards,
Matti
How will I make use of the updated plugin? (since this is included in fluentd's core).
I requested to @repeatedly that this change should be included in td-agent 3.5.1.
td-agent 3.5.1 will be published in the near future.