doublecloud/transfer

dst table was truncated during transfer migration

Closed this issue · 1 comments

I was migrating SNAPSHOT_AND_INCREMENT pg -> ch transfer from DoubleCloud to k8s with id preservation.

spec

transfer_spec:
  id: dttlkh8khl5e34ee9g3p
  type: SNAPSHOT_AND_INCREMENT
  src:
    type: pg
    params:
      Hosts:
        - ${DATATRANSFER_SRC_HOSTNAME}
      Port: 5432
      User: ${DATATRANSFER_SRC_USERNAME}
      Password: ${DATATRANSFER_SRC_PASSWORD}
      Database: xxxx
      EnableTLS: true
  dst:
    type: ch
    params:
      ShardsList:
        - Hosts:
            - ${DATATRANSFER_DST_HOSTNAME}
      User: ${DATATRANSFER_DST_USERNAME}
      Password: ${DATATRANSFER_DST_PASSWORD}
      Database: yyyy
      SSLEnabled: true
      HTTPPort: 8443
      NativePort: 9440
      Cleanup: Truncate
      Interval: 1000000000
      RetryCount: 20
      UseSchemaInTableName: true
      InsertParams:
        MaterializedViewsIgnoreErrors: true
  data_objects:
    include_objects:
      - public.history

Replication slot in PG was preserved and reused, but initial snapshot was also done, reread ~2M rows.

Logs from snapshot pod:

run activate with: *s3coordinator.CoordinatorS3

coordinator/coordinator_fake_client.go:96 fake change status: dttlkh8khl5e34ee9g3p -> Started

slot dttlkh8khl5e34ee9g3p in database xxxx exist: true

replication slot already exists, try to drop it

slot dttlkh8khl5e34ee9g3p in database xxxx exist: true

Will try to delete slot

Drop slot query executed

Slot should be deleted, double check

slot dttlkh8khl5e34ee9g3p in database support_chat exist: false

a replication slot already exists, Will sleep 677.78146ms and then retry create replication slot because of an error.

slot dttlkh8khl5e34ee9g3p in database xxxx exist: true

Will try to delete slot

Drop slot query executed

Slot should be deleted, double check

slot dttlkh8khl5e34ee9g3p in database support_chat exist: false

a replication slot already exists, Will sleep 677.78146ms and then retry create replication slot because of an error.

slot dttlkh8khl5e34ee9g3p in database xxxx exist: false

Create slot

Replication slot created, re-check existence

slot dttlkh8khl5e34ee9g3p in database xxxx exist: true

need to cleanup (Truncate) tables

Incremental pod (from sts) also contains logs about recreating replication slot and truncating table