airbytehq/terraform-provider-airbyte

Gsheet: updated connection returns 0 row

yamhirotoCS opened this issue · 0 comments

using provider 0.3.7.

Whenever I update some variables related to a gsheet source.
there isn't any Terraform fail : resource and connection get updated successfully.
but the sync returns 0.
The only way to make it work is to first get it destroyed (by commenting my code) and recreate it from scratch. Then it will sync data correctly.

Here is the log I get, I can see the right row counts but it doesn't do anything right after.

2023-12-08 09:42:29 source > Row counts: {'MY_GSHEET': 2128, 'Importrange_References': 4}
2023-12-08 09:42:29 source > Finished syncing spreadsheet 12hW9u2QSLNjwdAdNoJWGSCMFaODArnEHBHi9STiodkY
2023-12-08 09:42:29 platform > Total records read: 0 (0 bytes)
2023-12-08 09:42:29 platform > Schema validation was performed to a max of 10 records with errors per stream.
2023-12-08 09:42:29 platform > readFromSource: done. (source.isFinished:true, fromSource.isClosed:false)
2023-12-08 09:42:29 platform > processMessage: done. (fromSource.isDone:true, forDest.isClosed:false)
2023-12-08 09:42:29 platform > thread status... heartbeat thread: false , replication thread: true
2023-12-08 09:42:29 platform > writeToDestination: done. (forDest.isDone:true, isDestRunning:true)
2023-12-08 09:42:29 platform > thread status... timeout thread: false , replication thread: true

whereas when I recreate it 👍

2023-12-08 02:09:07 source > Starting syncing spreadsheet 12hW9u2QSLNjwdAdNoJWGSCMFaODArnEHBHi9STiodkY
2023-12-08 02:09:08 source > Available sheets: ['MY_GSHEET', 'Importrange_References']
2023-12-08 02:09:10 source > Row counts: {'MY_GSHEET': 2128, 'Importrange_References': 4}
2023-12-08 02:09:10 source > Syncing sheet MY_GSHEET
2023-12-08 02:09:11 source > Fetching range MY_GSHEET!2:202
2023-12-08 02:09:13 source > Fetching range MY_GSHEET!203:403
2023-12-08 02:09:14 source > Fetching range MY_GSHEET!404:604
2023-12-08 02:09:15 destination > 2023-12-08 02:09:15 INFO i.a.c.i.b.IntegrationCliParser(parseOptions):126 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2023-12-08 02:09:16 source > Fetching range MY_GSHEET!605:805
2023-12-08 02:09:16 destination > 2023-12-08 02:09:16 INFO i.a.c.i.b.a.AdaptiveDestinationRunner$Runner(getDestination):79 - Running destination under deployment mode: OSS
2023-12-08 02:09:17 destination > 2023-12-08 02:09:16 INFO i.a.c.i.b.a.AdaptiveDestinationRunner$Runner(run):100 - Starting destination: io.airbyte.integrations.destination.snowflake.SnowflakeDestination
2023-12-08 02:09:17 source > Fetching range MY_GSHEET!806:1006
2023-12-08 02:09:17 destination > 2023-12-08 02:09:17 INFO i.a.c.i.b.IntegrationCliParser(parseOptions):126 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2023-12-08 02:09:17 destination > 2023-12-08 02:09:17 INFO i.a.c.i.b.IntegrationRunner(runInternal):132 - Running integration: io.airbyte.integrations.destination.snowflake.SnowflakeDestination
2023-12-08 02:09:17 destination > 2023-12-08 02:09:17 INFO i.a.c.i.b.IntegrationRunner(runInternal):133 - Command: WRITE
2023-12-08 02:09:17 destination > 2023-12-08 02:09:17 INFO i.a.c.i.b.IntegrationRunner(runInternal):134 - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
2023-12-08 02:09:19 source > Fetching range MY_GSHEET!1007:1207
2023-12-08 02:09:19 destination > 2023-12-08 02:09:19 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword pattern_descriptor - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-12-08 02:09:19 destination > 2023-12-08 02:09:19 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-12-08 02:09:19 destination > 2023-12-08 02:09:19 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-12-08 02:09:19 destination > 2023-12-08 02:09:19 WARN i.a.c.i.b.DestinationConfig(initialize):36 - Singleton was already initialized.
2023-12-08 02:09:20 source > Fetching range MY_GSHEET!1208:1408
2023-12-08 02:09:21 source > Fetching range MY_GSHEET!1409:1609
2023-12-08 02:09:23 source > Fetching range MY_GSHEET!1610:1810
2023-12-08 02:09:24 source > Fetching range MY_GSHEET!1811:2011
2023-12-08 02:09:24 source > Finished syncing spreadsheet 12hW9u2QSLNjwdAdNoJWGSCMFaODArnEHBHi9STiodkY
2023-12-08 02:09:27 platform > Total records read: 1615 (964 KB)
2023-12-08 02:09:27 platform > Schema validation was performed to a max of 10 records with errors per stream.
2023-12-08 02:09:27 platform > readFromSource: done. (source.isFinished:true, fromSource.isClosed:false)
2023-12-08 02:09:27 platform > thread status... heartbeat thread: false , replication thread: true

To reproduce my error:

  • create a gsheet source + connection with the terraform provider
  • update names_conversion = false to names_conversion=true
  • apply the update
  • try to sync.

It's might be related to this error as well :
#31