cloudwatch log group not destroyed

[ferrandinand]

Terraform Version
Terraform v0.9.4

Affected Resource(s)
cloudwatch_log_group

Terraform Configuration Files
resource "aws_cloudwatch_log_group" "vpcflow_log_group" {
name = "vpcflow_log_group"
retention_in_days = "${var.vpc_flow_logs_retention}"
}

Expected Behavior
aws_cloudwatch_log_group should be removed when performing a destroy

Actual Behavior
Terraform destroy is showing that resource is destroyed but actually is still present.

Steps to Reproduce
$ terraform destroy
aws_cloudwatch_log_group.vpcflow_log_group: Refreshing state... (ID: vpcflow_log_group) aws_cloudwatch_log_group.vpcflow_log_group: Destroying... (ID: vpcflow_log_group) aws_cloudwatch_log_group.vpcflow_log_group: Destruction complete

Got to aws console/aws cli to check it is still there
aws logs describe-log-groups

[Graham-M]

I'm delighted you added this, I've exactly the same issue with VPC flow logs (although lagging on v0.9.3).

This is my suspicion:

• I believe there's some buffering that goes on with flushing of the logs from VPC flow logs to CloudWatch.
• When you delete the log group, there's still some logs still to be flushed from the buffer.
• I have an IAM permission applied to the VPC Flow logs that allows the CreateLogGroup permission.
• Once you delete the log group, a further flush happens, this permission kicks back in, thus recreating the log group.

Possible fix, if it's applicable to you:

--- iam.tf.old	2017-05-23 10:06:55.754475510 +0000
+++ iam.tf	2017-05-23 09:01:21.268106866 +0000
@@ -109,21 +109,20 @@
 resource "aws_iam_role_policy" "flowlogs_to_cloudwatch_policy" {
   name = "flowlogs-to-cloudwatch-policy-${var.vpc_id}"
   role = "${aws_iam_role.flowlogs_to_cloudwatch_role.id}"

   policy = <<EOF
 {
   "Version": "2012-10-17",
   "Statement": [
     {
       "Action": [
-        "logs:CreateLogGroup",
         "logs:CreateLogStream",
         "logs:PutLogEvents",
         "logs:DescribeLogGroups",
         "logs:DescribeLogStreams"
       ],
       "Effect": "Allow",
       "Resource": "*"
     }
   ]
 }
EOF
}

You create and destroy that log group yourself presumably, so why the need to allow VPC Flow Logs to also create and delete it? Hope it works for you too!

[ferrandinand]

Hi Graham,

Thanks for answering.
Currently we are already using the same policy you suggested but seems not to fix the issue.

data "aws_iam_policy_document" "flow_logs_policy" {
 statement {
   actions = [ 
     "logs:CreateLogGroup",
     "logs:CreateLogStream",
     "logs:PutLogEvents",
     "logs:DescribeLogGroups",
     "logs:DescribeLogStreams"
   ]   

   resources = [ 
     "*" 
   ]   
 }
}

[Graham-M]

Hi @ferrandinand, does it fix it if you remove the "logs:CreateLogGroup", permission?

[ferrandinand]

Hi @Graham-M
Sounds weird, because we need logs:CreateLogGroup permissions to create this, if not it won't be able to do it in terraform apply. isn't it?

How is this managed?

Thanks!

[Graham-M]

Hi @ferrandinand, it looks like you explicitly create the aws_cloudwatch_log_group yourself in your terraform, which is therefore managed by the user which executes terraform apply.

All the IAM role which you give to the Flow Logs needs to do is to be able to write to that aforementioned log group, therefore it just needs CreateLogStream and PutLogEvents to write to the group.

[catsby]

Hey @ferrandinand – I'm not able to reproduce this unfortunately. Are you able to capture any DEBUG logs (running terraform with TF_LOG=trace or similar) and see anything that raises suspicions regarding destroying the log group?

I was apple to apply and destroy an example log group like you showed with no error, and it was removed. I'm curious if this only happens in a certain situation.

thanks!

[Graham-M]

Hi @catsby,

I'm able to reproduce, but I don't think it's a terraform issue, but more due to the nature of AWS and its APIs. The debug log would simply show the log group being deleted and a clean exit from the destroy.

The issue would manifest the next time you came to apply. The log group that you were trying to create would already exist due to logs being buffered and flushed by Flow Logs/CloudWatch, which had am IAM permission to recreate the log group as soon as you destroyed it manually.

This needs to be undertaken in an environment with a busy VPC, it won't manifest in a quiet one with no flow logs to be flushed post-deletion.

Additionally, the workaround I suggested in my first reply does work here. If you create the log group yourself explicitly via conf and apply, the IAM requirement for Flow Logs to be able to CreateLogGroup is superfluous.

This is where it's recommended, and where @ferrandinand and I most likely copied and pasted from in the first instance:
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html#flow-logs-iam

[catsby]

Ah I see, thanks for pointing that out 😄
I'm going to close this for now, I agree it doesn't seem to be a Terraform issue.

Thanks!

[brandoncole]

Nice @Graham-M! Helped me too!

[androzo]

I had the same issue and removing the "logs:CreateLogGroup" from the policy resolved the issue in my case, thanks @Graham-M !

[Deleted user]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.