Dataset tables list update forces existing tables recreation with defined encryption

[hayorov]

Problem statement

Any dataset table list modification (e.g. new table, modification, or removal operation) triggers all exiting table recreation.

Environment

terraform 0.14
module 4.x/5
encryption defined (with dataset encryption_key).

Steps to reproduce

1. Define dataset with a single table A and some encryption key
2. Apply changes
3. Append table B with no other changes to tables A / dataset.
4. Run plan
5. Table A planned to be recreated

Plan output fragment

Terraform will perform the following actions:

  # module.project_bigquery["my_bq_dataset"].google_bigquery_table.main["a"] must be replaced
-/+ resource "google_bigquery_table" "main" {
      ~ creation_time       = 1620270233757 -> (known after apply)
      ~ etag                = "45iAIgM/cjwITp8LOlZQlA==" -> (known after apply)
      ~ id                  = "projects/X/datasets/my_bq_dataset/tables/a" -> (known after apply)
      ~ last_modified_time  = 1620270233818 -> (known after apply)
      ~ location            = "asia-southeast2" -> (known after apply)
      ~ num_bytes           = 0 -> (known after apply)
      ~ num_long_term_bytes = 0 -> (known after apply)
      ~ num_rows            = 0 -> (known after apply)
      ~ self_link           = "https://bigquery.googleapis.com/bigquery/v2/projects/X/datasets/my_bq_dataset/tables/a" -> (known after apply)
      ~ type                = "TABLE" -> (known after apply)
        # (7 unchanged attributes hidden)

      - encryption_configuration { # forces replacement
          - kms_key_name = "projects/Y/locations/asia-southeast2/keyRings/x-az1/cryptoKeys/default" -> null
        }

      ~ time_partitioning {
          ~ expiration_ms            = 0 -> (known after apply)
            # (2 unchanged attributes hidden)
        }
    }

  # module.project_bigquery["my_bq_dataset"].google_bigquery_table.main["b"] will be created
  + resource "google_bigquery_table" "main" {
      + clustering          = []
      + creation_time       = (known after apply)
      + dataset_id          = "my_bq_dataset"
      + etag                = (known after apply)
      + expiration_time     = 2524604400000
      + friendly_name       = "b"
      + id                  = (known after apply)
      + last_modified_time  = (known after apply)
      + location            = (known after apply)
      + num_bytes           = (known after apply)
      + num_long_term_bytes = (known after apply)
      + num_rows            = (known after apply)
      + project             = "X"
      + schema              = jsonencode(
            [
              + {
                  + description = "CLNTPFX"
                  + mode        = "REQUIRED"
                  + name        = "CLNTPFX"
                  + type        = "STRING"
                },
            ]
        )
      + self_link           = (known after apply)
      + table_id            = "b"
      + type                = (known after apply)

      + time_partitioning {
          + expiration_ms            = (known after apply)
          + require_partition_filter = false
          + type                     = "DAY"
        }
    }

Plan: 2 to add, 0 to change, 1 to destroy.

Expected behavior

Table B will be created with no impact to Table A

Root cause

The encryption attribute is defined by design as a dataset and manages (enforce) encryption for all tables. Table terraform resource itself has encryption attribute. Initial table creation inherits encryption value from a dataset but any next terraform action will set table encryption to "null", encryption key change forces table recreation and data loss.