aws_api_gateway_deployment doesn't get updated after changes

[blalor]

aws_api_gateway_deployment doesn't get updated after a dependent resource changes. For example, if I change a aws_api_gateway_integration resource to modify the request_template property, aws_api_gateway_deployment should be triggered. Since that doesn't happen, the stage specified in the deployment continues to use the old configuration. depends_on doesn't seem to be sufficient; I tried capturing that dependency and it didn't work, and as I understand it, depends_on only captures ordering.

A workaround is to taint the aws_api_gateway_deployment resource after a successful apply and re-running apply.

[jantman]

This is pretty important to me.

The issue seems to be that a deployment isn't really a resource on the AWS side; it functions like a logical deployment. When you create a deployment, it makes the current state of your REST API active; it's essentially a "commit" operation on the current configuration.

As a result:

1. The deployment needs to be "triggered" after any changes are made to the REST API to make them active.
2. The deployment must be after all other changes are completed in AWS, or else you'll end up deploying a partial API.

My workaround for this is as follows:

1. set the aws_api_gateway_deployment resource to have depends_on every other resource required for the API Gateway (i.e. all aws_api_gateway_* resources in the config).
2. as @blalor said, taint the deployment resource

[OJFord]

This is pretty critical - basically means aws_api_gateway_* isn't automated at all after the first run.

Further, setting depends_on = ["aws_api_gateway_integration.id"] doesn't work if it's only the included files (i.e. request_templates that change. I think that in itself is worth tracking separately: #8099.

TL;DR you can't rely on depends_on to make this work!

[apparentlymart]

Hi all,

The inability for Terraform to understand implicit update relationships between resources is known, and so far we've tended to solve it in more targeted ways by adding special "change detector" attributes to resources, such as the source_code_hash on aws_lambda_function, the etag on aws_s3_bucket_object, and the keepers on the resources in the random provider.

API Gateway doesn't make such a quick workaround simple because there isn't a good single thing we can take a hash of in order to represent a change.

However, one possible semi-fix we could do in the mean time is to add a triggers map to aws_api_gateway_deployment (like we have on null_resource) where you can interpolate items from your configuration that you know will change each time a significant change happens to your API. For example, if you're generating the API definitions from a Swagger file then you could put the hash of that file in the triggers. Terraform would then know that each time there is any change to a member of the triggers map it needs to create a new deployment, but it would be up to the user to define what exactly needs to change in order for that to happen.

I understand that such a solution is non-ideal due to the fact that there are so many little details in an API Gateway REST API, but I wonder if it would fill this hole slightly in the mean time. What do you think?

In the long run, something like what I proposed in #6810 could address this in a more robust way, though as proposed it would require a specific event annotation on every single API resource, which is not super-convenient. It would be nice to be able to easily specify the supposed common case of "re-deploy this API for any change to it".

[Ninir]

@apparentlymart The trigger seems a good idea :)
However, re-deploying the whole API for any change to it may not be the ideal idea, since you can try some resources from the console (without the need to deploy the API).

[OJFord]

re-deploying the whole API for any change to it may not be the ideal idea, since you can try some resources from the console (without the need to deploy the API)

You would probably only ever want that for e.g. "dev" stages though; so you could just define triggers = [] or whatever to disable it on those resources.

[sebolabs]

we have the same problem with our APIGs, dependencies just don't work at all
IMO either the aws_api_gateway_deployment resource should be redeployed on any change in methods, integrations, responses OR I would expect to have an attrib like force_redeploy that could be set to true and that would take care of this on demand

anyone figured out how to force redeployment with use of aws cli?

[Deleted user]

Couldn't all these problems be effectively solved (or rather, a workaround could be baked into the system) if the suggested triggers idea (or something similar) was added to everything, in a similar way to how depends_on is available more or less everywhere?

Something which says "if anything in any of these resources is being added/changed/destroyed then this resource needs to be updated". That would allow any and all of these implicit ordering issues to be addressed generically, rather than having to have issues filed and special solutions found for each thing. Perhaps with an option to say don't do it if any of those referenced things failed, which would preserve the intention of things like API Gateway deployments, where you're not meant to deploy something you know is broken.

Specific to API Gateway, what would probably work best from an ergonomic point of view would be a collection of stages attached to the aws_api_gateway_rest_api resource, and a way for Terraform to realise that something had changed related to that API and cause a redeploy to those stages after all those other things had been done (and only if they succeeded, too). This would help to avoid leaving out dependencies by having Terraform figure them out itself, but it doesn't fit the model of anything else Terraform does, you'd need the ability to defer part of a resource's execution to happen after some other set of things, which sounds like a distinctly non-trivial addition. Even more so than putting triggers (or whatever you might call it) on everything.

But without something, managing API Gateway using Terraform is always going to be problematic. Manual tainting of resources should be for exceptional cases, not everyday API updates.

[coryodaniel]

I ran into the same problem, but I'm not sure its such a bad thing. If I have multiple stages, I may not want to immediately release a deployment to prod just because I'm changing my integrations, methods, etc for testing on stage

The way I got around it was adding a deployed_at variable to my deployment and bumping it...

resource "aws_api_gateway_deployment" "instance" {
  rest_api_id = "${var.rest_api_id}"
  stage_name  = "${var.stage_name}"

  variables {
    deployed_at = "${var.deployed_at}"
  }
}

export TF_VAR_deployed_at=$(date +%s)
terraform plan

[claytono]

@coryodaniel I tried something similar:

resource "aws_api_gateway_deployment" "deployment" {
  rest_api_id = "${aws_api_gateway_rest_api.api.id}"
  stage_name  = "api"
  variables = {
    deployed_at = "${timestamp()}"
  }
}

I was hoping this would just force a redeploy, instead of deleting and recreating like a taint does. Unfortunately both taint and a variable change deletes and recreates with Terraform 0.9.1

[koenijn]

@ClaytonONeill I noticed that the stage_description field is also destroying and creating the deployment (TF 0.9.1). But if you look in the console it's actually creating a new deployment which you can see on the deployment history tab. This also means the stage id's and urls remain valid. So even if terraforms logging reports a full destroy it's not.

But I'm still struggling with the deployment process the API gateway provides supporting multiple environments (stages?) from one API definition and integrating it in our other terraform scripting which creates a single environment. This would mean duplicate API definitions in the API gateway, but we only need a deployment per environment. Maybe we can switch off the creation of the definitions using count = ${var.develop_environment} and only create the deployment or so.

[claytono]

@koenijn you're 100% correct. I've updated my resource to look like this:

resource "aws_api_gateway_deployment" "deployment" {
  depends_on = ["aws_api_gateway_method.method"]

  rest_api_id = "${aws_api_gateway_rest_api.api.id}"
  stage_name  = "api"
  description = "Deployed at ${timestamp()}"
}

With this approach it forces a new deploy every time I run apply, which is a decent work around for the time being.

[koenijn]

@ClaytonONeill the description attribute doesn't work for me, it just updates the existing deployment without adding the new/changes resources. It does for you? But the stage_description does:

resource "aws_api_gateway_deployment" "ApiDeployment" {
  depends_on = [
    "aws_api_gateway_integration.PostRegisterIntegration",
    "aws_api_gateway_integration.GetLicenseIntegration"
  ]

  rest_api_id = "${aws_api_gateway_rest_api.MyApi.id}"
  stage_name  = "${var.apex_environment}"

  stage_description = "${timestamp()}" // forces to 'create' a new deployment each run
  description = "Deployed at ${timestamp()}" // just some comment field which can be seen in deployment history

  variables = {
    "lambdaAlias" = "${var.apex_environment}"
  }
}

[claytono]

@koenijn Tested again, and you're right again. I originally set mine to stage_description and validated that worked. Then thought it'd be nice to use description instead, but didn't test more than once. I've switched back to stage_description.

[charlieegan3]

I solved this using a hash of the file that described the gateway resources.

resource "aws_api_gateway_deployment" "default" {
  ...
  stage_description = "${md5(file("api_gateway.tf"))}"
  ...
}

Not sure if it's a good idea but it seems to work better than the timestamp option.

[aterreno]

Nice solution @charlieegan3 - that's the only sensible way of working around this TF bug.

[szilveszter]

I have tried the workarounds mentioned above, but I keep getting the following error (using TF v0.10.6):

Error applying plan:

1 error(s) occurred:

* module.api.aws_api_gateway_deployment.deployment (destroy): 1 error(s) occurred:

* aws_api_gateway_deployment.deployment: BadRequestException: Active stages pointing to this deployment must be moved or deleted
	status code: 400, request id: a3f7493d-bd8d-11e7-94ef-87cf3d993ad2

Here's my resource definition:

resource "aws_api_gateway_deployment" "deployment" {
  depends_on = ["aws_api_gateway_rest_api.api"]

  rest_api_id = "${aws_api_gateway_rest_api.api.id}"
  stage_name  = "${var.stage}"

  variables {
    deployed_at = "${timestamp()}"
  }
}

Did I miss anything from the helpful comments above, or does this simply not work with more recent versions of TF?

[aterreno]

From your error message I think the error is different this time, you still have APIs deployed there, in stages and AWS won't let you delete (you are doing a destroy right?) if that's the case.

You need to 'empty' the gateway first and then destroy with TF.

[szilveszter]

@aterreno I got the error message during an apply. The first run created everything successfully, then I kicked off another apply to see if I get a new deployment, but I got the error above.

[Puneeth-n]

@szilveszter Had this issue today in prod. We don't update our APIG so often. The trick is to use aws_api_gateway_deployment to deploy to an intermediate stage and use that deployment id to deploy to the actual stage

I had the same issue today and fixed it like I mentioned. To verify, check the deployment timestamp of the final stage.

# Deploy each time
# https://github.com/hashicorp/terraform/issues/6613
# Since we don't use templates to deploy APIG (Unlike step functions), we cannot redeploy on change in APIG definition,
# Let's redeploy each time.
resource "aws_api_gateway_deployment" "currency-deployment" {
    rest_api_id = "${aws_api_gateway_rest_api.currency.id}"
    stage_name  = "intermediate"
    stage_description = "Deployed at: ${timestamp()}"

    depends_on  = ["aws_api_gateway_integration.get-rates-integration", "aws_api_gateway_method.get-rates-method", "aws_api_gateway_integration_response.get-rates-method-integration-200", "aws_api_gateway_method_response.get-rates-method-200"]
}

resource "aws_api_gateway_stage" "stage" {
    stage_name  = "${var.environment}"
    rest_api_id = "${aws_api_gateway_rest_api.currency.id}"
    deployment_id = "${aws_api_gateway_deployment.currency-deployment.id}"
}

resource "aws_api_gateway_base_path_mapping" "custom-domain" {
    api_id      = "${aws_api_gateway_rest_api.currency.id}"
    stage_name  = "${var.environment}"
    domain_name = "${var.domain_name}"
    base_path   = "${aws_api_gateway_rest_api.currency.name}"

    depends_on = ["aws_api_gateway_stage.stage"]
}

[gregor2004ua]

Bouncing off of @koenijn and @charlieegan3 I also added the ignore_changes attribute to my deployment to avoid terraform changing description on the deployment every time I run it.

resource "aws_api_gateway_deployment" "default" {
  ...
  stage_description = "${md5(file("api_gateway.tf"))}"
  description = "Deployed at ${timestamp()}"
  lifecycle = {
      ignore_changes = ["description"]
  }
  ...
}

[sebnyberg]

Since I define my api across several files in my module, I found it useful to define a computed local which combines the hashes of different files like so:

locals {
  file_hashes = [
    "${md5(file("${path.module}/api_gateway.tf"))}",
    "${md5(file("${path.module}/users.tf"))}",
    "${md5(file("${path.module}/posts.tf"))}",
  ]
  combined_hash = "${join(",", local.file_hashes)}"
}

resource "aws_api_gateway_deployment" "example" {
  ...
  stage_description = "${local.combined_hash}"
}

That way, a new deployment will happen whenever any of the files are changed.

[sebolabs]

@sebnyberg I'm using locals too but instead of having two locals I have only one that looks like this:

  stage_description = "${md5(
    format("%s%s",
      file("${path.module}/resource-proxy.tf"),
      file("${path.module}/resource-root.tf"),
    )
  )}"

[tdmalone]

I'm running into the same issue as @szilveszter, but I think it's because we're potentially using the idea of stages and deployments incorrectly. The deployment works great the first time, but after that it can't be destroyed because stages are pointing at it. I can't easily use @Puneeth-n's solution, because our API and it's two stages (dev and prod) are created in one Terraform state, with other state files 'injecting' their own endpoints into the API. We therefore want to force redeployments, from these other state files, when changes are made - without access to change the original stage.

Going from the idea of deployments being like 'commits', it seems there's an inherent incompatibility between deployments and the Terraform model anyway, which works best with 'resources'.

So, my solution for now. It's likely this will become a pain if I have to do it often, but at the moment when I need to force a redeploy I am removing the existing deployments from the state:

$ terraform state rm aws_api_gateway_deployment.dev
$ terraform state rm aws_api_gateway_deployment.prod
$ terraform apply

This simply forces Terraform to 'forget' about the existing deployments and create new ones, which you pretty much don't want to do for normal resources... but it seems to me to fit in this situation (and no, it doesn't seem necessary to update the deployment_id set on the stage, I imagine having the stage_name set in the deployment is what is taking care of that).

[Puneeth-n]

@tdmalone if in the deployment resource, if you pass empty stage_name, the stage won't be created. Only deployment will be done and then use that deployment id to created your stages. Thus you don't need an intermediate stage.

The solution I suggested is old. Checkout my new module and how I do it:

https://github.com/comtravo/terraform-aws-api-gateway/blob/23dfbda3d027e03e6f186bb36989797a192bdd02/main.tf#L9-L19

[nguse]

Using stage_description seems like a good work-around to solve this issue.

Rather than just using the md5 of a file or something, I'm putting the ids of all dependent resources in the stage_description, so it's a little more targeted:

resource "aws_api_gateway_deployment" "example" {
  rest_api_id = "${aws_api_gateway_rest_api.example.id}"
  stage_name  = "test"

  # Force re-deployments if any dependencies change
  # https://github.com/hashicorp/terraform/issues/6613
  stage_description = <<DESCRIPTION
${aws_api_gateway_resource.example.id}
${aws_api_gateway_method.example.id}
${aws_api_gateway_integration.example.id}
DESCRIPTION

  depends_on = [
    "aws_api_gateway_integration.example",
  ]
}

[blalor]

I ran into this problem again today when using aws_api_gateway_base_path_mapping. The following gave me problems when updating the aws_api_gateway_integration, with the error:

* aws_api_gateway_deployment.main: BadRequestException: Active stages pointing to this deployment must be moved or deleted

resource "aws_api_gateway_integration" "example" {
  rest_api_id = "${local.rest_api_id}"

  resource_id = "${aws_api_gateway_resource.example.id}"
  http_method = "${aws_api_gateway_method.example.http_method}"

  type = "HTTP_PROXY"
  integration_http_method = "ANY"
  uri = "https://example.com/{version}.tar.gz"

  request_parameters = {
    "integration.request.path.version" = "method.request.path.version"
  }

  passthrough_behavior = "WHEN_NO_MATCH"
  cache_key_parameters = [
    "method.request.path.version",
  ]
}

resource "random_pet" "deployment_trigger" {
  length = 3

  keepers = {
    gw_int_example_uri                     = "${aws_api_gateway_integration.example.uri}"
    gw_int_example_integration_http_method = "${aws_api_gateway_integration.example.integration_http_method}"
    gw_int_example_request_params          = "${jsonencode(aws_api_gateway_integration.example.request_parameters)}"
  }
}

resource "aws_api_gateway_deployment" "main" {
  depends_on = [
    "aws_api_gateway_integration.example",
  ]

  rest_api_id = "${local.rest_api_id}"

  stage_name = "prod"
  stage_description = "${random_pet.deployment_trigger.id}"
}

resource "aws_api_gateway_base_path_mapping" "example" {
  domain_name = "${aws_api_gateway_domain_name.main.domain_name}"
  base_path = "example"

  api_id     = "${local.rest_api_id}"
  stage_name = "${aws_api_gateway_deployment.main.stage_name}"
}

To resolve it, I changed the aws_api_gateway_deployment to have stage_name = "${random_pet.deployment_trigger.id}". Not particularly clean, but it works.

[neg3ntropy]

Best that I have got:

resource "aws_api_gateway_deployment" "deployment" {
  rest_api_id = "${aws_api_gateway_rest_api.api.id}"
  stage_name  = ""
  stage_description = "Terraform hash ${md5(join(",", local.all_resources))}"
  
  lifecycle {
    create_before_destroy = true
  }
}

locals {
  all_resources = [
    "${aws_api_gateway_resource.mount_resource.path}:${aws_api_gateway_method.mount_get.http_method}:${aws_api_gateway_integration.mount_get_integration.uri}",
  ]
}

Seems to work for most updates, but there are serious design issues in the deployement "resource".

[flintinatux]

Varying the stage_description is what finally worked for me. All the other methods resulted in errors and a failed apply. This is what I landed on:

resource "aws_api_gateway_deployment" "service" {
  depends_on = [
    "aws_api_gateway_integration.lambda",
    "aws_api_gateway_integration.lambda_root",
  ]

  rest_api_id       = "${aws_api_gateway_rest_api.service.id}"
  stage_description = "${md5(file("${path.module}/api_gateway.tf"))}"
  stage_name        = "${var.env}"

  lifecycle {
    create_before_destroy = true
  }
}

[udondan]

Thanks for all the comments here. That helped a lot. Based on that I came up with something similar to redeploy whenever my local IP changes during testing.

data "http" "icanhazip" {
  url = "http://icanhazip.com"
}

locals {
  whitelist_ips = [
    "${data.http.icanhazip.body}",
    "some.static.ip",
    "another.static.ip",
  ]

  whitelist_ips_hash = "${md5(join(",", local.whitelist_ips))}"
}

resource "aws_api_gateway_deployment" "..." {
  ...
  variables {
    whitelist_ips_hash = "${local.whitelist_ips_hash}"
  }
}

[apparentlymart]

Hi all!

This issue was migrated over to hashicorp/terraform-provider-aws#162 as part of splitting the providers into their own repositories. It looks like the bot that did it got its commenting privileges rate limited during the migration and so it unfortunately didn't post a specific note about it here at the time.

The resource in question here is no longer in this repository, so please take further discussion on this issue over to hashicorp/terraform-provider-aws#162 where the AWS provider maintainers can see it.

Thanks!