aws-samples/amazon-ecs-firelens-examples

parse ecs nginx logs with fluentbit

anton-demydov-zoral opened this issue · 6 comments

anton-demydov-zoral commented

Hello @PettitWesley. Can you please advice here
I have such task definition

[{
		"essential": true,
		"image": "AWS_ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/custom-fluent-bit:latest",
		"name": "log_router",
		"firelensConfiguration": {
			"type": "fluentbit",
			"options": {
				"enable-ecs-log-metadata": "false"
			}
		},
		"logConfiguration": {
			"logDriver": "awslogs",
			"options": {
				"awslogs-group": "${aws_cloudwatch_log_group.log-group.name}",
				"awslogs-region": "us-east-1",
				"awslogs-create-group": "true",
				"awslogs-stream-prefix": "firelens"
			}
		},
		"memoryReservation": 50
	},
	{
		"name": "${local.env}-${local.application}",
		"image": "${var.repo_url}:${var.image_tag}",
		"essential": true,
		"portMappings": [{
			"containerPort": 80
		}],
		"logConfiguration": {
			"logDriver": "awsfirelens",
			"options": {
				"Name": "es",
				"Host": "${var.elasticsearch_host}",
				"Port": "443",
				"Index": "my_index",
				"Type": "my_type",
				"Aws_Region": "us-east-1",
				"tls": "On"
			}
		},
		"memory": 512,
		"cpu": 256
	}
]

I'm preparing custom custom-fluent-bit images.
Dockerfile:

FROM amazon/aws-for-fluent-bit:latest
ADD fluent-bit.conf /fluent-bit/etc/
ADD parsers.conf /fluent-bit/parsers/

I need to parse nginx logs, so here's my fluent-bit.conf:

[SERVICE]
    Parsers_File /fluent-bit/parsers/parsers.conf
    Log_Level debug

[INPUT]
    Name forward
    unix_path /var/run/fluent.sock

[FILTER]
    Name parser
    Match **
    Parser nginx
    Key_Name log

[OUTPUT]
    Name es
    Match *
    Host  ES_HOST
    Port  443
    Index my_index
    Type  my_type

but the problem is that I don't see parsed nginx log. Instead of this I see something like:

{
  "_index": "my_index",
  "_type": "my_type",
  "_id": "C8H09nUBFsE_GWj4U09x",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-11-23T21:13:25.050Z",
    "container_id": "94ebad2b6cfd8a078f776ac8f0947ecb072e111db34ebe24d699a2ca9e29d208",
    "container_name": "/ecs-dev-emulator-sdk-35-dev-service-f4e6ceceadcfd4819401",
    "source": "stdout",
    "log": "172.16.6.223 - - [23/Nov/2020:21:13:25 +0000] \"GET / HTTP/1.1\" 200 5901 \"-\" \"ELB-HealthChecker/2.0\" \"-\""
  },
  "fields": {
    "@timestamp": [
      "2020-11-23T21:13:25.050Z"
    ]
  },
  "sort": [
    1606166005050
  ]
}

Can you please advice how to parse line "log": "172.16.6.223 - - [23/Nov/2020:21:13:25 +0000] \"GET / HTTP/1.1\" 200 5901 \"-\" \"ELB-HealthChecker/2.0\" \"-\"" ?

anton-demydov-zoral commented

Also, there is a problem with config-file-type:

 {
        "essential": true,
        "image": "829824778702.dkr.ecr.us-east-1.amazonaws.com/custom-fluent-bit:latest",
        "name": "log_router",
        "firelensConfiguration": {
                "type": "fluentbit",
                "options":{
                  "enable-ecs-log-metadata":"false",
                  "config-file-type": "file",
                  "config-file-value": "/extra.conf"
            }
        },

extra.conf:

[FILTER]
    Name parser
    Match **
    Parser nginx
    Key_Name log

I've tried to use latest, 2.2.0, 2.3.0 containers from amazon/aws-for-fluent-bit but no luck.
If I use latest - fluentbit doesn't even start:

[2020/11/24 11:21:38] [error] [lib] backend failed 

If I use 2.2.0:

[2020/11/24 10:58:02] [error] [filter_parser] requested parser 'nginx' not found

I've tried to add parsers.conf but it didn't help as well.

PettitWesley commented

Sorry I missed this, this example should work: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/parse-common-log-formats

Your extra.conf is missing the Service section to import parsers: https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/mainline/examples/fluent-bit/parse-common-log-formats/parse-apache.conf

That's all you should need. You should not add your own input in the extra config file.

anton-demydov-zoral commented

Thanks @PettitWesley . I'll check and let you know.

anton-demydov-zoral commented

Hey @PettitWesley
It seems I still see the issue with parsing. Please check my configuration.
Dockerfile:

FROM amazon/aws-for-fluent-bit:latest
ADD fluent-bit.conf /fluent-bit/etc/

fluent-bit.conf:

[SERVICE]
    Parsers_File /fluent-bit/parsers/parsers.conf
    Log_Level debug

[INPUT]
    Name forward
    unix_path /var/run/fluent.sock

[FILTER]
    Name parser
    Match **
    Parser nginx
    Key_Name log
    Reserve_Data True

[OUTPUT]
    Name es
    Match *
    Host  AWS_ES_HOST
    Port  443
    Index react-sdk
    Type  my_type

I can confirm that I see unparsed logs in elasticsearch:
Screenshot 2020-12-21 at 13 43 50

but I assume fluentbit should parse them to something like:

{
  "remote": "172.16.5.61",
  "host": "-",
  "user": "-",
  "method": "GET",
  "path": "/",
  "code": "200",
  "size": "5901",
  "referer": "-",
  "agent": "ELB-Healthchecker"
}
PettitWesley commented

@anton-demydov-zoral Sorry, with the holidays I was on vacation.

Please follow this example for adding a custom config file: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/config-file-type-file

There are two problems with yours:

  1. You use the /fluent-bit/etc path for your config, which is the path used by FireLens for its generated config, you need to use a different file path. This should be noted in our documentation.
  2. You should not add the input for logs, FireLens will take care of that with the managed config. Your extra config is imported into the managed config.

Here is what you should have:

Dockerfile:

FROM amazon/aws-for-fluent-bit:latest
ADD extra.conf /extra.conf

Config in extra.conf:

[SERVICE]
    Parsers_File /fluent-bit/parsers/parsers.conf
    Log_Level debug

[FILTER]
    Name parser
    Match **
    Parser nginx
    Key_Name log
    Reserve_Data True

[OUTPUT]
    Name es
    Match *
    Host  AWS_ES_HOST
    Port  443
    Index react-sdk
    Type  my_type

Use the link I provided to see what the Task Definition should look like, there are two key sections. On your Fluent Bit container you need to have:

"firelensConfiguration": {
				"type": "fluentbit",
				"options": {
					"config-file-type": "file",
					"config-file-value": "/extra.conf"
				}
			},

And then in your app container that produces logs you should have:

"logConfiguration": {
				 "logDriver":"awsfirelens",
}
  • ❤️1
anton-demydov-zoral commented

Finally, it works, thank you @PettitWesley !
Screenshot 2021-01-05 at 17 59 28

just posting here my configs for future needs.
Dockerfile:

FROM amazon/aws-for-fluent-bit:latest
ADD extra.conf /extra.conf

extra.conf:

[SERVICE]
    Log_Level info
    Parsers_File /fluent-bit/parsers/parsers.conf
[FILTER]
    Name parser
    Match **
    Parser nginx
    Key_Name log
    Reserve_Data True
[OUTPUT]
    Name es
    Match *
    Host  AWS_ES_HOST
    Port  443
    Index MY_INDEX
    Type  my_type
    tls     On
    tls.verify Off
    Aws_Region us-east-1

and Task Definition:

[
    {
	"essential": true,
	"image": "REPO",
	"name": "log_router",
	"firelensConfiguration": {
		"type": "fluentbit",
                "options":{
                  "enable-ecs-log-metadata": "false",
                  "config-file-type": "file",
                  "config-file-value": "/extra.conf"
            }
	},
	"logConfiguration": {
		"logDriver": "awslogs",
		"options": {
			"awslogs-group": "log_group",
			"awslogs-region": "us-east-1",
			"awslogs-create-group": "true",
			"awslogs-stream-prefix": "firelens"
		}
	},
	"memoryReservation": 50
	},
    {
      "name": "name",
      "image": "image_name}",
      "essential": true,
      "portMappings": [
        {
          "containerPort": 80
                }
      ],
      "logConfiguration": {
                "logDriver": "awsfirelens"
            },
      "memory": 512,
      "cpu": 256
    }
  ]