aws-powertools/powertools-lambda-python

Bug: Conflict between POWERTOOLS_LOG_LEVEL and Lambda applicationLogLevelV2

Closed this issue · 12 comments

andresionek91 commented

Expected Behaviour

POWERTOOLS_LOG_LEVEL (and POWERTOOLS_LOGGER_SAMPLE_RATE`) should decide what log lines Powertools emits.

Lambda’s applicationLogLevelV2 should only act as a platform filter (drop logs below that threshold before forwarding to CloudWatch).

Examples:

  • If POWERTOOLS_LOG_LEVEL=INFO and applicationLogLevelV2=DEBUG, then only INFO+ should be emitted, and those should appear in CloudWatch.
  • If POWERTOOLS_LOG_LEVEL=DEBUG and applicationLogLevelV2=INFO, then DEBUG logs should be emitted by Powertools but dropped by the platform, so only INFO+ appear in CloudWatch.

Current Behaviour

applicationLogLevelV2=DEBUG causes all logs, including DEBUG, to appear in CloudWatch, even if POWERTOOLS_LOG_LEVEL=INFO.

In effect, the Lambda application log level is overriding Powertools’ log level instead of filtering logs after emission.

POWERTOOLS_LOGGER_SAMPLE_RATE becomes ineffective when combined with applicationLogLevelV2=DEBUG, because all requests in a container emit DEBUG regardless of Powertools’ base level.

Code snippet

from aws_lambda_powertools import Logger

logger = Logger()

@logger.inject_lambda_context
def handler(event, context):
    logger.debug("Debug log line")
    logger.info("Info log line")
    return {"status": "ok"}

Possible Solution

Ensure Powertools filtering (POWERTOOLS_LOG_LEVEL, POWERTOOLS_LOGGER_SAMPLE_RATE) applies before logs are emitted.

The Lambda runtime should then filter logs again according to applicationLogLevelV2.

If this behavior is by design, document clearly that:

  • applicationLogLevelV2=DEBUG will force DEBUG logs through, regardless of Powertools config.
  • Sampling (POWERTOOLS_LOGGER_SAMPLE_RATE) works per-container and will appear overridden if applicationLogLevelV2=DEBUG is set.

Steps to Reproduce

Case 1

  • applicationLogLevelV2 = INFO
  • POWERTOOLS_LOG_LEVEL = DEBUG
  • Behaviour: Only INFO+ appear.
  • ✅ Expected: Only INFO+ appear. (works correctly)

Case 2

  • applicationLogLevelV2 = DEBUG
  • POWERTOOLS_LOG_LEVEL = INFO
  • Behaviour: DEBUG logs also appear.
  • ❌ Expected: Only INFO+ logs, since Powertools should suppress DEBUG.

Case 3

  • applicationLogLevelV2 = DEBUG
  • POWERTOOLS_LOG_LEVEL = INFO
  • POWERTOOLS_LOGGER_SAMPLE_RATE = 0.5
  • Behaviour: All requests show DEBUG logs.
  • ❌ Expected: Only ~50% of requests should show DEBUG logs.

Case 4

  • applicationLogLevelV2 = INFO
  • POWERTOOLS_LOG_LEVEL = INFO
  • POWERTOOLS_LOGGER_SAMPLE_RATE = 0.5
  • Behaviour: Only INFO+ appear.
  • ❌ Expected: Some requests/containers show DEBUG logs due to sampling.

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.12

Packaging format used

PyPi

Debugging logs

boring-cyborg commented 
Thanks for opening your first issue here! We'll come back to you as soon as we can.

In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link
leandrodamascena commented 
Hi @andresionek91, thanks for opening this issue. This is an interesting report on how Powertools behaves when you set applicationLogLevelV2 and the Powertools log level. Let me go trough some points from your comments to explain how this works.




Current Behaviour


applicationLogLevelV2=DEBUG causes all logs, including DEBUG, to appear in CloudWatch, even if POWERTOOLS_LOG_LEVEL=INFO.


In effect, the Lambda application log level is overriding Powertools’ log level instead of filtering logs after emission.




This is exactly the behavior we expect here, because since Lambda's log level works at the ControlPlane level and takes precedence over any configuration you have in your application, then in this line we check if AWS_LAMBDA_LOG_LEVEL is set. If so, we intentionally set the Powertools log level to the one defined in the AWS_LAMBDA_LOG_LEVEL environment variable.


In this line we also raise a warning to let developers know that there is difference between log levels and it can lead to data loss.




POWERTOOLS_LOGGER_SAMPLE_RATE becomes ineffective when combined with applicationLogLevelV2=DEBUG, because all requests in a container emit DEBUG regardless of Powertools’ base level.




Yes, you're right. Since we'll be adjusting the log level to the one defined in AWS_LAMBDA_LOG_LEVEL, and this is debug, the sampling rate won't work because all debug logs will be flushed. I don't exactly your use case, but probably make sense you take a look at Log Buffer feature here: https://docs.powertools.aws.dev/lambda/python/latest/core/logger/#buffering-logs




Possible Solution


Ensure Powertools filtering (POWERTOOLS_LOG_LEVEL, POWERTOOLS_LOGGER_SAMPLE_RATE) applies before logs are emitted.


The Lambda runtime should then filter logs again according to applicationLogLevelV2.




We don't expect to change this behavior in Powertools, but of course we are always listening to customer/community feedback. If we see a good amount of customers requesting these changes we may consider a change in the next major release.




If this behavior is by design, document clearly that:


    

  • applicationLogLevelV2=DEBUG will force DEBUG logs through, regardless of Powertools config.
    • 

  • Sampling (POWERTOOLS_LOGGER_SAMPLE_RATE) works per-container and will appear overridden if applicationLogLevelV2=DEBUG is set.
    • 





We provide documentation with a diagram to explain how this works. In this documentation, we also have a section explaining the priority order:




We prioritise log level settings in this order:






AWS_LAMBDA_LOG_LEVEL environment variable

Explicit log level in Logger constructor, or by calling the logger.setLevel() method

POWERTOOLS_LOG_LEVEL environment variable




But now I wonder if this isn't clear enough and if we need to improve. I'd love to hear your thoughts on this and perhaps you can send a PR to help improve it.


Thank you again for reporting this, and please let me know if you need any further help.
andresionek91 commented 
@leandrodamascena Thanks for the detailed explanation — I completely missed that part of the documentation (even ChatGPT missed it when I asked whether there were any documented conflicts between log level settings 😅).


Part of my confusion came from using AWS CDK and setting the applicationLogLevelV2 parameter. It wasn’t transparent to me that this was setting the AWS_LAMBDA_LOG_LEVEL environment variable behind the scenes. To make things trickier, the Lambda console doesn’t display AWS_LAMBDA_-prefixed env vars either, so the override wasn’t obvious.


Suggestions:


    

  1. 

    Add a warning callout in the [sampling section](https://docs.powertools.aws.dev/lambda/python/latest/core/logger/#sampling-logs) explaining that setting applicationLogLevelV2 (via CDK or console) will override Powertools’ log level and sampling. You could anchor-link to the priority diagram you mentioned.
    

    2. 

  2. 

    While this is arguably more of a CDK/Lambda console issue, the lack of transparency was the root of my confusion. A short note that “setting applicationLogLevelV2 will automatically set the hidden AWS_LAMBDA_LOG_LEVEL env var” would have saved me a lot of time. Do you think it’s worth adding that clarification?
    

    3. 

  3. 

    Consider extending the existing diagram to make the relationship explicit:
    

    ApplicationLogLevelV2 (CDK/Console)
         ↓
Sets AWS_LAMBDA_LOG_LEVEL env var
         ↓
Overrides Powertools log level (and sampling)

    

    4. 



Workaround I ended up with:


    

  • Set LoggingFormat to TEXT. With JSON format, CDK requires applicationLogLevelV2, which implicitly sets AWS_LAMBDA_LOG_LEVEL.
    • 



I'm happy to contribute with a PR to update the docs if you think it's worth mentioning it.
leandrodamascena commented 


@leandrodamascena Thanks for the detailed explanation — I completely missed that part of the documentation (even ChatGPT missed it when I asked whether there were any documented conflicts between log level settings 😅).




Ah, I see. Yeah, sometimes models can't detect everything. But if you're using GenAI and Powertools to build your workloads, I suggest using our MCP server, which can be quite helpful because it read our documentation and help a lots. https://github.com/aws-powertools/powertools-mcp




Suggestions:


    

  1. Add a warning callout in the [sampling section](docs.powertools.aws.dev/lambda/python/latest/core/logger#sampling-logs) explaining that setting applicationLogLevelV2 (via CDK or console) will override Powertools’ log level and sampling. You could anchor-link to the priority diagram you mentioned.
    2. 





Yes, that makes a lot of sense. I don't necessarily like the term applicationLogLevelV2 because it's too programmatic and not everyone is using CDK/Cloudformation. I prefer to use Advanced Logging controls and we can link to CDK/API. What do you think?




    

  1. While this is arguably more of a CDK/Lambda console issue, the lack of transparency was the root of my confusion. A short note that “setting applicationLogLevelV2 will automatically set the hidden AWS_LAMBDA_LOG_LEVEL env var” would have saved me a lot of time. Do you think it’s worth adding that clarification?
    2. 





Do you mean to explain that Lambda sets this env var when using Advanced Logging control? Yep, I think it make sense.




    

  1. Consider extending the existing diagram to make the relationship explicit:>    ```>    ApplicationLogLevelV2 (CDK/Console)
    
↓
    
Sets AWS_LAMBDA_LOG_LEVEL env var
    
↓
    
Overrides Powertools log level (and sampling)

    

    

    2. 





I think it can hard represent in a diagram, but we can try.




Workaround I ended up with:


    

  • Set LoggingFormat to TEXT. With JSON format, CDK requires applicationLogLevelV2, which implicitly sets AWS_LAMBDA_LOG_LEVEL.
    • 



I'm happy to contribute with a PR to update the docs if you think it's worth mentioning it.




Sure sure, please submit a PR and lets work together to get it merged.


Thanks a lot.
dreamorosi commented 
Hi @andresionek91, are you still interested in sending a PR to fix this section of the docs?


If priorities have changed we understand and it's more than fine; in that case we'll put back the issue in the backlog and open it for other contributors.
leandrodamascena commented 
For someone that is reading this and wants to work in this issue:


• Fix broken link in docs/core/logger.md (line ~378) - replace with correct AWS Lambda log level documentation URL


• Add warning callout in the sampling section explaining how Advanced Logging controls overrides Powertools for AWS settings


• Clarify CDK behavior - explain that applicationLogLevelV2 constructor automatically sets AWS_LAMBDA_LOG_LEVEL env var


• Enhance priority section - make it clear that Lambda's log level takes precedence over Powertools for AWS logger config


Any questions please ask here.
  • 👍1
powertools-for-aws-oss-automation commented 
Note
For those interested in contributing, please leave a comment below so that we can assign the issue to you and make sure we don't duplicate efforts. If you have any further questions, please don't hesitate to ask below or on our Discord server.



 
kattakaha commented 
Hi @leandrodamascena @dreamorosi,


I'd like to work on this issue! I've already implemented the documentation improvements as outlined in the requirements:


✅ Fixed broken link - Updated the AWS Lambda Advanced Logging Controls link to point to the correct documentation

✅ Added warning in sampling section - Explained how ALC with DEBUG level overrides sampling behavior

✅ Clarified CDK behavior - Added info box explaining how applicationLogLevelV2 automatically sets AWS_LAMBDA_LOG_LEVEL

✅ Enhanced priority section - Made it clear with concrete examples how Lambda's log level takes precedence


The changes include:


    

  • Fixed link on line ~378 to correct AWS documentation URL
    • 

  • Added warning callout in sampling section about ALC interaction
    • 

  • Added info box in priority section explaining CDK/console behavior with concrete example
    • 



I'll prepare a PR shortly. Thank you for the clear guidance on what needed to be addressed!
  • 👍1
dreamorosi commented 
Great, thank you!


Please make sure to set up your local development environment as mentioned in the CONTRIBUTING.md doc and run the local make pr command before pushing.


If you have any questions, please let us know, thank you again!
powertools-for-aws-oss-automation commented 
Warning
This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either reopen the issue, or open a new issue referencing this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.