Is it possible to define custom input tag for fluentbit configs?
bhargavamin opened this issue · 4 comments
Hi,
I have usecase where I'm moving my infrastructure to ECS Fargate and out like to define custom input configs.
In my task definition, I'm launching two containers: app and fluentbit. The app container uses firelens to forward stdout logs to fluentbit, which then forwards the logs to fluentd aggregators.
Additionally, I create a custom output config file at /fluent-bit/configs/custom-output.conf in fluentbit image with the following content:
[OUTPUT]
Name forward
Match **
Host FLUENT_AGGREGATOR_URL
Port FLUENT_AGGREGATOR_PORT
tls on
tls.verify off
The Host and Port are replaced using entrypoint.sh script(both are passed to task definition as environment variables).
The logs that get forwarded have a customer index/tag which starts with service name defined in task definition.
Task definition
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
- Name: nginx-web
Cpu: !Ref ContainerCpu
Essential: true
Image: !Ref ContainerImageTag
Memory: !Ref ContainerMemory
PortMappings:
- ContainerPort: !Ref ContainerPort
LogConfiguration:
LogDriver: awsfirelens
- Name: fluentbit-agent
Image: !Ref FluentbitImageUrl
Essential: true
Environment:
- Name: "FLUENT_AGGREGATOR_URL"
Value: !Ref FluentdAggregatorUrl
- Name: "FLUENT_AGGREGATOR_PORT"
Value: !Ref FluentdAggregatorPort
FirelensConfiguration:
Type: fluentbit
Options:
config-file-type: "file"
config-file-value: "fluent-bit/configs/custom-output.conf"
LogConfiguration:
LogDriver: awslogs
Options :
awslogs-group: !Ref CloudWatchLogsGroup
awslogs-region: !Ref AWS::Region
awslogs-create-group: true
awslogs-stream-prefix: nginx-web/fluentbit-logs
MemoryReservation: 50
Cpu: !Ref ContainerCpu
ExecutionRoleArn: !GetAtt TaskExecutionRole.Arn
Memory: !Ref Memory
NetworkMode: awsvpc
Family: webapp
RequiresCompatibilities:
- FARGATESample log
{
"_index": "nginx-web-45",
"_type": "fluentd",
"_id": "EVLFrXUBCNxJLISHC",
"_version": 1,
"_score": null,
"_source": {
"container_name": "/ecs-nginx-web-TaskDefinition-19BCASD9W3QZ-1-nginx-web-d0bcb99c14300",
"source": "stdout",
"log": "10.10.37.65 - - [09/Nov/2020:16:09:46 +0000] \"GET / HTTP/1.1\" 200 2350 \"-\" \"ELB-HealthChecker/2.0\"",
"container_id": "add7cee329be065b3735eb0ba8b8",
"ecs_cluster": "arn:aws:ecs:eu-west-1:323423439477:cluster/web",
"ecs_task_arn": "arn:aws:ecs:eu-west-1:3234234239477:task/web/808ec9a8fc7b68b895f03c95",
"ecs_task_definition": "nginx-web:16",
"@timestamp": "2020-11-09T16:09:46.000000000+00:00"
},
"fields": {
"@timestamp": [
"2020-11-09T16:09:46.000Z"
]
},
"sort": [
1604938186000
]
}
From the above example, the logs are sent with Tag as service namenginx-web-45.
So the question here is:
Is possible to also have a custom input config? if yes, is it possible to have custom input Tag?
Any guidance would be much appreciated.
thanks in advance!
The tag is set by FireLens and is always in the format of {container name}-firelens-{task ID}. There is no way to change this directly.
You can however use Fluent Bit plugins to change the tag after the input stage. You can use a filter to change the tag: https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag
Fluentd has a rewrite tag filter as well, so you could also change the tag in your Fluentd aggregator.
I think Wesley's suggestion for using rewrite-tag can be workaround for this. As there is no activity for a long time, I am closing this. Feel free to reopen if you still have questions.
The tag is set by FireLens and is always in the format of {container name}-firelens-{task ID}. There is no way to change this directly.
You can however use Fluent Bit plugins to change the tag after the input stage. You can use a filter to change the tag: https://docs.fluentbit.io/manual/pipeline/filters/rewrite-tag
Fluentd has a rewrite tag filter as well, so you could also change the tag in your Fluentd aggregator.
is this information in any documentation ? It would help me a lot to know where to find this kind of information.