s3 source requesting deprecated provider
LeopoldGabelmann opened this issue · 2 comments
Hi there,
I am trying to add a s3 source with the terraform airbyte provider (v0.3.4) on our airbyte instance (v0.50.30). For the source I am trying to run this code:
terraform {
required_providers {
airbyte = {
source = "airbytehq/airbyte"
version = "0.3.4"
}
}
}
provider "airbyte" {
server_url = "http://localhost:8006/v1/"
}
resource "airbyte_source_s3" "test_source_s3_cac" {
configuration = {
aws_access_key_id = "<access_key>"
aws_secret_access_key = "<secret>"
bucket = "<bucket_name>"
dataset = "template_cac_source_s3"
path_pattern = "folder/*"
source_type = "s3"
format = {
source_s3_update_file_format_csv = {
block_size = 10000
delimiter = ","
double_quote = true
encoding = "utf8"
escape_char = ""
filetype = "csv"
infer_datatypes = true
newlines_in_values = false
quote_char = "\""
}
}
provider = {
aws_access_key_id = "<access_key>"
aws_secret_access_key = "<secret>"
path_prefix="folder/*"
bucket = "<bucket_name>"
}
streams = [ {
file_type = "parquet"
name = "template_cac_source_s3"
} ]
}
name = "template_source.cac.cac_test"
workspace_id = var.test_workspace
}
resource "airbyte_destination_snowflake" ... {
...
}
resource "airbyte_connection" "test_snowflake_connection_cac" {
source_id = airbyte_source_s3.test_source_s3_cac.source_id
destination_id = airbyte_destination_snowflake.test_destination_snowflake_cac.destination_id
}
In the terraform documentation, they state that the provider is deprecated, however, running it without raises the error:
│ {"type":"https://reference.airbyte.com/reference/errors#unprocessable-entity","title":"unprocessable-entity","status":422,"detail":"The provided configuration does not fulfill the specification. Errors: json schema
│ validation failed when comparing the data to the json schema. \nErrors: $.provider: is missing but it is required "}
Running it with the provider causes the connection to raise the error:
│ {"type":"https://reference.airbyte.com/reference/errors","title":"unexpected-problem","status":400,"detail":"Something went wrong in the connector. logs:init() got an unexpected keyword argument 'bucket'"}
what probably is caused by the deprecated provider. Any clue what's going on here?
have you updated your connector on Airbyte ? I had the same issue at first, then I upgraded S3 connector to 4.0.4 and I don't get this issue anymore
Sorry for my long absence, yes that was the case and thus this is solved. Thanks @yamhirotoCS