Support custom connectors
aballiet opened this issue · 13 comments
A lot of Airbyte users implement custom connectors using python CDK or low-code CDK.
Currently it's not possible to configure sources leveraging custom connector.
It will be nice to have source_generic and destination_generic resources.
In the end I made a Terraform provider for Airbyte OSS version which supports custom connectors.
You can find a full example here if interested 👍
@aballiet looks nice your work. I'll try to use it
we were waiting so long for this.
Thanks
@aballiet looks nice your work. I'll try to use it
we were waiting so long for this.
Thanks
Yes it's frustrating...
Tell me if you experience any issue 😊
@aballiet looks nice your work. I'll try to use it
we were waiting so long for this.
ThanksYes it's frustrating...
Tell me if you experience any issue 😊
i'll keep you update
wondering if I can add custom destination with your repo ?
Because we are using airbyte as reverse ETL and airbyte is not ready for this so we keep it only as custom destination
@aballiet looks nice your work. I'll try to use it
we were waiting so long for this.
ThanksYes it's frustrating...
Tell me if you experience any issue 😊i'll keep you update wondering if I can add custom destination with your repo ? Because we are using airbyte as reverse ETL and airbyte is not ready for this so we keep it only as custom destination
I think you can using this : https://registry.terraform.io/providers/aballiet/airbyte-oss/latest/docs/resources/airbyte_destination_definition
@aballiet +1 nice work. I'm in the early phases on testing your provider and it's looking promising.
It's unfortunate the official provider doesn't support this yet, Airbyte recently added the ability to create custom connectors in the Airbyte API - https://reference.airbyte.com/reference/createsource
@aballiet +1 amazing work.
Is this compatible with sources created using Connector Builder Intro or it's YAML
or this only targeted for connectors created using the CDK.
In a more philosophical question, would you want your code to be part of the official terraform-airbyte provider? or is there any reason you prefer not to.
@aballiet +1 nice work. I'm in the early phases on testing your provider and it's looking promising.
It's unfortunate the official provider doesn't support this yet, Airbyte recently added the ability to create custom connectors in the Airbyte API - https://reference.airbyte.com/reference/createsource
@stevenmurphy12 this only allows to create a source from an already defined connector in the instance, isn't it?
Allows for creation of source but not the underlying connector for that source.
@NumberPiOso - Yes that's correct.
Being able to create the actual connector definition was not a particularly high priority for me, though I was able to use @aballiet provider to do that - https://registry.terraform.io/providers/aballiet/airbyte-oss/latest/docs/resources/airbyte_source_definition.
I'm currently trying out a hybrid. Using the airbyte-oss provider for:
- Source connector definition
- Source creation
And using airbytehq/airbyte for:
- Destination creation (I'm using one of the standard destinations)
- Connection creation.
@aballiet +1 amazing work. Is this compatible with sources created using Connector Builder Intro or it's YAML or this only targeted for connectors created using the CDK.
In a more philosophical question, would you want your code to be part of the official terraform-airbyte provider? or is there any reason you prefer not to.
It's working for both. We build connector using low-code (YAML) and build docker image with it. See here for details, it will generate you a Dockerfile when selecting on the following :
The option ❯ Configuration Based Source - Generate a Source that is described using a low code configuration file.
Then you can use your YAML manifest to build the connector, push it to your artifact registry and use the Terraform provider with it 👍
Does anybody know wether there is an official solution for a source definition?