airbytehq/terraform-provider-airbyte

s3 source requesting deprecated provider

LeopoldGabelmann opened this issue · 2 comments

Hi there,
I am trying to add a s3 source with the terraform airbyte provider (v0.3.4) on our airbyte instance (v0.50.30). For the source I am trying to run this code:

terraform {
  required_providers {
    airbyte = {
      source = "airbytehq/airbyte"
      version = "0.3.4"
    }
  }
}


provider "airbyte" {
  server_url = "http://localhost:8006/v1/"
}


resource "airbyte_source_s3" "test_source_s3_cac" {
  configuration = {
    aws_access_key_id = "<access_key>"
    aws_secret_access_key = "<secret>"
    bucket = "<bucket_name>"
    dataset = "template_cac_source_s3"
    path_pattern = "folder/*"
    source_type = "s3"
    format = {
      source_s3_update_file_format_csv = {
        block_size = 10000
        delimiter = ","
        double_quote = true
        encoding = "utf8"
        escape_char = ""
        filetype = "csv"
        infer_datatypes = true
        newlines_in_values = false
        quote_char = "\""
      }
    }
    provider = {
      aws_access_key_id = "<access_key>"
      aws_secret_access_key = "<secret>"
      path_prefix="folder/*"
      bucket = "<bucket_name>"
    }
    streams = [ {
      file_type = "parquet"
      name = "template_cac_source_s3"
    } ]
  }
  name         = "template_source.cac.cac_test"
  workspace_id = var.test_workspace
}


resource "airbyte_destination_snowflake" ... {
...
}


resource "airbyte_connection" "test_snowflake_connection_cac" {
  source_id = airbyte_source_s3.test_source_s3_cac.source_id
  destination_id = airbyte_destination_snowflake.test_destination_snowflake_cac.destination_id
}

In the terraform documentation, they state that the provider is deprecated, however, running it without raises the error:

│ {"type":"https://reference.airbyte.com/reference/errors#unprocessable-entity","title":"unprocessable-entity","status":422,"detail":"The provided configuration does not fulfill the specification. Errors: json schema
│ validation failed when comparing the data to the json schema. \nErrors: $.provider: is missing but it is required "}

Running it with the provider causes the connection to raise the error:

│ {"type":"https://reference.airbyte.com/reference/errors","title":"unexpected-problem","status":400,"detail":"Something went wrong in the connector. logs:init() got an unexpected keyword argument 'bucket'"}

what probably is caused by the deprecated provider. Any clue what's going on here?

have you updated your connector on Airbyte ? I had the same issue at first, then I upgraded S3 connector to 4.0.4 and I don't get this issue anymore

Sorry for my long absence, yes that was the case and thus this is solved. Thanks @yamhirotoCS