Control plane restart

[fulmicoton]

Observed on cicada : On control plane restart, upon reception of the chitchat state, we end up removing a lot of shards.

2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.820Z  INFO quickwit_control_plane::model: removing shards from model source_uid=app-logs-649:01HP6FTGHCS28TTAN2SMJPP9FV:_ingest-source shard_ids=[ShardId("01HVRBQ92NRTPBHCV4V7VKBK3Z")]
2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.820Z  INFO quickwit_control_plane::control_plane: received eof shard via gossip shard_id=01HVRKG1BCBBKV5KVWWQYC1P52 position=Position::Eof
2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.820Z  INFO quickwit_control_plane::control_plane: deleting shards shard_ids=[ShardId("01HVRKG1BCBBKV5KVWWQYC1P52")] source_uid=SourceUid { index_uid: IndexUid { index_id: "app-logs-346", incarnation_id: Ulid(2064201455267077262770700750118270741) }, source_id: "_ingest-source" }
2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.823Z  INFO quickwit_control_plane::model: removing shards from model source_uid=app-logs-346:01HP6FTGE8NG862NPA8J06ADRN:_ingest-source shard_ids=[ShardId("01HVRKG1BCBBKV5KVWWQYC1P52")]
2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.823Z  INFO quickwit_control_plane::control_plane: received eof shard via gossip shard_id=01HVRKG1BCBF76A11B833W3SX8 position=Position::Eof
2024-04-18T12:42:53Z app[e82d41ec03e498] iad [info]2024-04-18T12:42:53.823Z  INFO quickwit_control_plane::control_plane: deleting shards shard_ids=[ShardId("01HVRKG1BCBF76A11B833W3SX8")] source_uid=SourceUid { index_uid: IndexUid { index_id: "app-logs-346", incarnation_id: Ulid(2064201455267077262770700750118270741) },