awslabs/aws-embedded-metrics-python

Not working with batch and fargate tasks

frediana opened this issue · 2 comments

Hello,

I successfully integrated aws-embedded-metrics-python with some lambda in my project. Works very well.

I have some issues though to get it work on AWS Batch that run using Fargate Tasks. These tasks are configured with a "awslog" LogDriver and because Fargate tasks does not support running a side-car cloudwatch agent I'm kind of stuck.

What I tried (but didn't work so far) is to set AWS_EMF_ENVIRONMENT=local environment variable on the Fargate task to force embeded metric to be written to stdout in cloudwatch log group (instead of trying use cloudwatch agent that does not running).

I can see the emf json is well written into the log group, and aws-embedded-metrics-python does not try to send metrics to cloudwatch agent.

Problem: This emf json metric is somehow not parsed correctly by Cloudwatch metric somehow., So event if those emf json are written, I cannot graph them from cloudwatch metrics.

I have a feeling the problem comes from logdriver="awslog" which may not handle emf format somehow properly?

Do you have any possible workaround for my situation?

Related to this maybe ? moby/moby#42731

@frediana, any EMF logs that can reach CloudWatch Logs will now be parsed and metrics can be generated from them. You don't need the EMF header anymore.

If you face any issues, you can get in touch with me!