terraform-google-modules/terraform-google-bigquery

range_partitioning does not work

halil-burak opened this issue · 4 comments

halil-burak commented

When we try to create our BQ tables, we want to use range_partitioning in one of them. We could not generate the range partition on of them. I've been using this module for a while but not able to use range_partitioning on a table. There is no error but the range partitioning is not applied, is not shown on plan output.

We are using this terraform registry and Google's provider for this.
https://registry.terraform.io/modules/terraform-google-modules/bigquery/google/latest

morgante commented

Can you share your Terraform config?

halil-burak commented

Hi,
Of course. I'm sharing the scripts and configs below.

module usage with dataset and table with range partitioning

module "testing_the_range_partitioning" {
  source  = "terraform-google-modules/bigquery/google"
  version = "~> 4.4"

  dataset_id   = "testing_the_range_partitioning"
  dataset_name = "test range"
  description  = "Dataset containing tables for person and job staging data"
  project_id   = var.project_id
  location     = "EU"

  tables = [
    {
      table_id            = "testing_the_range_partitioning_table",
      schema              = "./schema.json",
      time_partitioning   = null,
      deletion_protection = false,
      range_partitioning = {
        field = "personId",
        range = {
          start    = "1",
          end      = "1000000",
          interval = "10000",
        },
      },
      expiration_time = null,
      clustering      = null,
      labels          = {},
    },
  ]
  access = [] # pass empty not to conflict with below
}

provider.tf for google provider --> Google as a provider

provider "google" {
  version     = "~> 3.90"
  project     = var.project_id
  region      = var.region
  credentials = file("key.json")
}

schema.json file

[
  {
    "mode": "NULLABLE",
    "name": "personId",
    "type": "INTEGER"
  },
  {
      "fields": [
      {
        "name": "actor",
        "type": "STRING"
      },
      {
        "name": "action",
        "type": "STRING"
      }
    ],
    "mode": "REPEATED",
    "name": "personData",
    "type": "RECORD"
  },
  {
    "mode": "NULLABLE",
    "name": "createdAt",
    "type": "TIMESTAMP"
  },
  {
    "mode": "NULLABLE",
    "name": "updatedAt",
    "type": "TIMESTAMP"
  }
  
]
halil-burak commented

I've figured out what was the issue. We were using version 4.3 and 4.4 of this module, I upgraded to version 5.X and started getting schema errors. After some research, I found out that I had to use file() function for the json schema. Then it worked.

morgante commented

@halil-burak Glad it worked for you!

  • 👍1