meltano/sdk

Improve the behavior of the standard load methods in targets

Opened this issue · 0 comments

Based on a discussion in target-postgres MeltanoLabs/target-postgres#266.

The current Singer convention in targets is to upsert records. If key properties are set then the target will update any records that exist instead of inserting.

Thoughts are it relates to the standard load_method configs:

  • Should the default actually be upsert? Or do we want to remove the default all together to make this config become more of an override to force the target to act a certain way vs a required value?
  • We should implement logic so that the target throws an error if someone selects upsert but theres no key properties available
  • If the default behavior of the SDK was always to upsert then we should change the logic to first check the load_method, if its set to append-only then dont upsert even if key properties are available.
  • Do existing targets get this functionality out of the box properly or is there anything they need to implement? If so, we should document it.