Reapply after destroy fails, because of Cloudwatch Loggroup already exists

Description

I got the following example Terraform code, see below. It applies perfectly the first time and I got the output displayed. It also destroys fine.
But when I try to reapply it a second time, it fails with the following error:
Error: Creating CloudWatch Log Group failed: ResourceAlreadyExistsException: The specified log group already exists: The CloudWatch Log Group '/aws/lambda/my-lambda1' already exists.

✋ I have searched the open/closed issues and my issue is not listed.

Versions

Module version [Required]: 3.3.1
Terraform version:
Terraform v1.2.2
on darwin_amd64
Provider version(s):
provider registry.terraform.io/hashicorp/aws v4.25.0
provider registry.terraform.io/hashicorp/external v2.2.2
provider registry.terraform.io/hashicorp/local v2.2.3
provider registry.terraform.io/hashicorp/null v3.1.1

Reproduction Code [Required]

Lambda example

def lambda_handler(event, context):
    message = 'Hello Dude, {} {}!'.format(event['first_name'], event['last_name'])  
    return { 
        'message' : message
    }

Terraform example

module "lambda_function" {
  source = "terraform-aws-modules/lambda/aws"

  publish = true

  function_name = "my-lambda-test"
  description   = "My awesome lambda function"
  handler       = "handler.lambda_handler"
  runtime       = "python3.9"

  source_path = "${path.module}/lambda-example"

  tags = local.common_tags
}

data "aws_lambda_invocation" "example" {
  function_name = module.lambda_function.lambda_function_name

  input = <<JSON
{
  "first_name": "Jo",
  "last_name": "Awesome"
}
JSON
  depends_on = [
    module.lambda_function
  ]
}

output "result" {
  value = data.aws_lambda_invocation.example.result
}

Steps to reproduce the behavior:

terraform init
terraform apply
terraform destroy
terraform apply

Expected behavior

It should apply everything at the second time without error.

Actual behavior

After destroying the example, something is left behind/recreated in AWS. Because next apply fails with following error.
Error: Creating CloudWatch Log Group failed: ResourceAlreadyExistsException: The specified log group already exists: The CloudWatch Log Group '/aws/lambda/my-lambda1' already exists.

Additional context

I think the "problem" arise since I use a aws_lambda_invocation data source to invoke the lambda right after deploy. This means that it will be invoked right before destroy, lambda output is buffered, Terraform destroys the Lambda and Cloudwatch loggroup and output is released to Cloudwatch afterwards and recreates the loggroup again.
This problem is also described here: hashicorp/terraform#14750
The last post refers to that the Lambda doesn't need to have the logs:CreateLogGroup permission, which resolves the issue.

I want to suggest to completely remove the logs:CreateLogGroup permission here

terraform-aws-lambda/iam.tf

Line 123 in 9d16478

!var.use_existing_cloudwatch_log_group ? "logs:CreateLogGroup" : "",

.
The loggroup gets created anyway by Terraform, either because this modules does it or the user does it himself and sets var.use_existing_cloudwatch_log_group=true.

Additionally, wouldn't it be better to pass an ARN to var.use_existing_cloudwatch_log_group instead of a boolean?
I was wondering how a custom managed log group should be named and the module currently assumes it's always /aws/lambda/<name-of-your-lambda.

Hi Johannes,

AWS Lambda service does not allow customization of the name of the log group (no ARN can be specified), so users should either have use_existing_cloudwatch_log_group = true to manage the log group outside of the module or let the module create log group.

I don't think there is anything we can do to handle CW log groups better in this module.

IIRC we had similar issues with the EKS module and the way log groups behave there. @bryantbiggs, what do you think?

Hi Anton,
I see, thanks for the explanation.
But at least I think the logs:CreateLogGroup permission is really unnecessary as stated, the log group will be either created by an TF resource of this module or something outside by the customer. In both ways the Lambda itself doesn't need to have the IAM permission, IMO.

IIRC without this permission, Lambda itself won't be able to create log group(s) in some cases (e.g. with Lambda@Edge, there is a log group per region).

I see, yes, Lambda@Edge would have problems then or the customer would need to create the log group in every region which is also cumbersome.
Hmmm ... probably the permission can be related to Lambda@Edge use cases only?!
Let's see what @bryantbiggs is thinking about it.

I worked around it now by managing the cloudwatch log group separately.

This issue case isn't isolated to @jgrumboe ;

I also manage to be blocked by this issue, but strangely after a destroy / manual deleting cloudwatch log group, trying multiple redeployment, it's gone... For now, I can't explain it
If you have a permanent fix avoiding us the destroy/manual deleting cloudwatch log group, it would be great

(PS : Thank you for this incredible module !)

Faced this issue recently, but in my case, I noticed log group is already deleted. I think somehow the module still presumes that log group is there even though it's not there.

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

This issue was automatically closed because of stale in 10 days

Is it fixed ? if not, please reopen the issue

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.