practo/tipoca-stream

Not required schema migration happening after using schema passed from batcher

alok87 opened this issue · 2 comments

alok87 commented 
I0409 19:14:56.032218       1 redshift.go:432] tOps: [ALTER TABLE "db"."table" ADD COLUMN  "plan_id" character varying(144)   ], cOps: [], vOps: []
I0409 19:14:56.032274       1 redshift.go:457] Strategy2: starting inplace-migration, table
I0409 19:14:56.294294       1 redshift.go:474] Running: ALTER TABLE "table"."col" ADD COLUMN  "plan_id" character varying(144)

Using the latest schema version is better than always doing migrations for each schema change. Assuming the latest schema is backward compatible makes things fast. This is for the first time sink case.

This issue was born after #189

alok87 commented
  • #188 makes the call to check schema exists only once for one topic, this is not causing this problem.
  • #189 is also not causing this.

Will come back to this.

alok87 commented

This can be solved by using latest schemaID instead of schemaID from the job.

schemaId = job.SchemaId

https://github.com/practo/tipoca-stream/blob/master/redshiftsink/pkg/redshiftloader/load_processor.go#L659-L663

Using latest schemaID always to make the redshift table should solve this problem. Later.