TestRestoreSchemaRoundtripIntegration is flaky
Closed this issue · 0 comments
karol-kokoszka commented
It fails randomly on different PRs and different env setups.
Example:
--- Run restore with properties: map[location:[s3:restoretest-schema-roundtrip] restore_schema:true snapshot_tag:sm_20240723145615UTC]
14:56:22.228 INFO restore Restore {"cluster_id": "b1bbe658-4903-11ef-85c2-6045bd04f3e6", "task_id": "b9c647b2-4903-11ef-85d9-6045bd04f3e6", "run_id": "b9c647b6-4903-11ef-85da-6045bd04f3e6"}
14:56:22.278 INFO client Checking hosts connectivity {"hosts": ["2001:db9:200::11", "2001:db9:200::12", "2001:db9:200::13", "2001:db9:200::21", "2001:db9:200::22", "2001:db9:200::23"]}
14:56:22.279 INFO client Host check OK {"host": "2001:db9:200::22"}
14:56:22.279 INFO client Host check OK {"host": "2001:db9:200::11"}
14:56:22.279 INFO client Host check OK {"host": "2001:db9:200::21"}
14:56:22.279 INFO client Host check OK {"host": "2001:db9:200::23"}
14:56:22.280 INFO client Host check OK {"host": "2001:db9:200::13"}
14:56:22.280 INFO client Host check OK {"host": "2001:db9:200::12"}
14:56:22.280 INFO client Done checking hosts connectivity
14:56:22.280 INFO restore No datacenter specified for location - using all nodes for this location {"location": "s3:restoretest-schema-roundtrip"}
14:56:22.312 INFO client Host location access check OK {"host": "2001:db9:200::23", "location": "s3:restoretest-schema-roundtrip"}
14:56:22.315 INFO client Host location access check OK {"host": "2001:db9:200::12", "location": "s3:restoretest-schema-roundtrip"}
14:56:22.316 INFO client Host location access check OK {"host": "2001:db9:200::22", "location": "s3:restoretest-schema-roundtrip"}
14:56:22.321 INFO client Host location access check OK {"host": "2001:db9:200::13", "location": "s3:restoretest-schema-roundtrip"}
14:56:22.321 INFO client Host location access check OK {"host": "2001:db9:200::11", "location": "s3:restoretest-schema-roundtrip"}
14:56:22.323 INFO client Host location access check OK {"host": "2001:db9:200::21", "location": "s3:restoretest-schema-roundtrip"}
14:56:22.323 INFO restore Found hosts with location access {"location": "s3:restoretest-schema-roundtrip", "hosts": ["2001:db9:200::11", "2001:db9:200::12", "2001:db9:200::13", "2001:db9:200::21", "2001:db9:200::22", "2001:db9:200::23"]}
14:56:22.323 INFO restore Look for schema file
14:56:22.328 INFO restore Found schema file
14:56:22.328 INFO restore Initialized target {"target": {"location":["s3:restoretest-schema-roundtrip"],"keyspace":["system_schema"],"snapshot_tag":"sm_20240723145615UTC","batch_size":2,"restore_schema":true,"continue":true}}
14:56:22.376 INFO restore Initialized units {"units": [{"keyspace":"system_schema","size":807379,"tables":[{"table":"columns","tombstone_gc":"timeout","size":147407},{"table":"computed_columns","tombstone_gc":"timeout","size":53026},{"table":"indexes","tombstone_gc":"timeout","size":45632},{"table":"tables","tombstone_gc":"timeout","size":116202},{"table":"scylla_tables","tombstone_gc":"timeout","size":61[69](https://github.com/scylladb/scylla-manager/actions/runs/10061207962/job/27810699498?pr=3938#step:4:70)5},{"table":"view_virtual_columns","tombstone_gc":"timeout","size":48902},{"table":"dropped_columns","tombstone_gc":"timeout","size":42503},{"table":"views","tombstone_gc":"timeout","size":41736},{"table":"keyspaces","tombstone_gc":"timeout","size":42359},{"table":"scylla_keyspaces","tombstone_gc":"timeout","size":39731},{"table":"functions","tombstone_gc":"timeout","size":34620},{"table":"aggregates","tombstone_gc":"timeout","size":34038},{"table":"types","tombstone_gc":"timeout","size":33390},{"table":"scylla_aggregates","tombstone_gc":"timeout","size":33114},{"table":"triggers","tombstone_gc":"timeout","size":33024}]}]}
14:56:22.401 INFO restore Initialized views {"views": null}
14:56:22.404 INFO restore Awaiting schema agreement...
14:56:22.409 INFO restore Done awaiting schema agreement {"duration": "5.6259[76](https://github.com/scylladb/scylla-manager/actions/runs/10061207962/job/27810699498?pr=3938#step:4:77)ms"}
14:56:22.409 INFO restore Started restoring schema
14:56:22.409 INFO restore Apply schema CQL statements
14:56:24.532 INFO restore Restoring schema finished
helper_integration_test.go:216: run restore: restore data: create tab_b1f65286490311ef85d06045bd04f3e6 (roundtrip_b1f65261490311ef85cf6045bd04f3e6) with CREATE TABLE roundtrip_b1f65261490311ef85cf6045bd04f3e6.tab_b1f65286490311ef85d06045bd04f3e6 (
id int,
data blob,
PRIMARY KEY (id)
) WITH ID = b2559f20-4903-11ef-87e6-83fe86068e27
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
AND comment = ''
AND compaction = {'class': 'NullCompactionStrategy', 'enabled': 'false'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND speculative_retry = '99.0PERCENTILE'
AND paxos_grace_seconds = 864000
AND tombstone_gc = {'mode': 'repair', 'propagation_delay_in_seconds': '3600'}: Cannot add already existing table "tab_b1f65286490311ef[85](https://github.com/scylladb/scylla-manager/actions/runs/10061207962/job/27810699498?pr=3938#step:4:86)d06045bd04f3e6" to keyspace "roundtrip_b1f65261490311ef85cf6045bd04f3e6"
--- FAIL: TestRestoreSchemaRoundtripIntegration (17.17s)
https://github.com/scylladb/scylla-manager/actions/runs/10061207962/job/27810699498?pr=3938