Storm: Network: Topology redeployment: ISL between deactivated switches is in DISCOVERED status (should be FAILED)
yuliiamir opened this issue · 1 comments
yuliiamir commented
The issue occurred when the network topology has been redeployed after killing the network topology with the following switches' deactivation. As a result, after recovering, both switches are marked as DEACTIVATED (expected behavior). BUT ISL between deactivated switches is in DISCOVERED status(incorrect behaviour). The rest of the links that are connected to the DEACTIVATED switches are in FAILED status.
Kibana logs:
ISL that should be in FAILED status(both switches are in DEACTIVATED status):
ISL is in FAILED status (one of the switches is in DEACTIVATED status):
Topology:
Links details after redeploying network topology(switches were switched off after network topology deletion):
[
{
"latency_ns": 3086294,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:07",
"port_no": 50,
"seq_id": 0,
"segment_latency": 3086294
},
{
"switch_id": "00:00:00:00:00:00:00:08",
"port_no": 7,
"seq_id": 1,
"segment_latency": 3086294
}
]
},
{
"latency_ns": 3659898,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:03",
"port_no": 4,
"seq_id": 0,
"segment_latency": 3659898
},
{
"switch_id": "00:00:00:00:00:00:00:04",
"port_no": 4,
"seq_id": 1,
"segment_latency": 3659898
}
]
},
{
"latency_ns": 3786802,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:08",
"port_no": 7,
"seq_id": 0,
"segment_latency": 3786802
},
{
"switch_id": "00:00:00:00:00:00:00:07",
"port_no": 50,
"seq_id": 1,
"segment_latency": 3786802
}
]
},
{
"latency_ns": 3197969,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:04",
"port_no": 4,
"seq_id": 0,
"segment_latency": 3197969
},
{
"switch_id": "00:00:00:00:00:00:00:03",
"port_no": 4,
"seq_id": 1,
"segment_latency": 3197969
}
]
},
{
"latency_ns": 3532994,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", // sw_02 is deactivated - OK
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 2,
"seq_id": 0,
"segment_latency": 3532994
},
{
"switch_id": "00:00:00:00:00:00:00:04",
"port_no": 1,
"seq_id": 1,
"segment_latency": 3532994
}
]
},
{
"latency_ns": 4992,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:03",
"port_no": 6,
"seq_id": 0,
"segment_latency": 4992
},
{
"switch_id": "00:00:00:00:00:00:00:08",
"port_no": 16,
"seq_id": 1,
"segment_latency": 4992
}
]
},
{
"latency_ns": 3340101,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", sw_02 is deactivated - OK
"actual_state": "FAILED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:04",
"port_no": 1,
"seq_id": 0,
"segment_latency": 3340101
},
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 2,
"seq_id": 1,
"segment_latency": 3340101
}
]
},
{
"latency_ns": 5069,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:08",
"port_no": 16,
"seq_id": 0,
"segment_latency": 5069
},
{
"switch_id": "00:00:00:00:00:00:00:03",
"port_no": 6,
"seq_id": 1,
"segment_latency": 5069
}
]
},
{
"latency_ns": 5399,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", sw_02 is deactivated - OK
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 3,
"seq_id": 0,
"segment_latency": 5399
},
{
"switch_id": "00:00:00:00:00:00:00:03",
"port_no": 1,
"seq_id": 1,
"segment_latency": 5399
}
]
},
{
"latency_ns": 3730964,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:08",
"port_no": 49,
"seq_id": 0,
"segment_latency": 3730964
},
{
"switch_id": "00:00:00:00:00:00:00:09",
"port_no": 52,
"seq_id": 1,
"segment_latency": 3730964
}
]
},
{
"latency_ns": 4587,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", sw_02 is deactivated - OK
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 1,
"seq_id": 0,
"segment_latency": 4587
},
{
"switch_id": "00:00:00:00:00:00:00:03",
"port_no": 2,
"seq_id": 1,
"segment_latency": 4587
}
]
},
{
"latency_ns": 4812,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", sw_02 is deactivated - OK
"actual_state": "FAILED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:03",
"port_no": 1,
"seq_id": 0,
"segment_latency": 4812
},
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 3,
"seq_id": 1,
"segment_latency": 4812
}
]
},
{
"latency_ns": 3583756,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:09",
"port_no": 52,
"seq_id": 0,
"segment_latency": 3583756
},
{
"switch_id": "00:00:00:00:00:00:00:08",
"port_no": 49,
"seq_id": 1,
"segment_latency": 3583756
}
]
},
{
"latency_ns": 6520,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", sw_02 is deactivated - OK
"actual_state": "FAILED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:03",
"port_no": 2,
"seq_id": 0,
"segment_latency": 6520
},
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 1,
"seq_id": 1,
"segment_latency": 6520
}
]
},
{
"latency_ns": 3446700,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", sw_01 is deactivated - OK
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:01",
"port_no": 2,
"seq_id": 0,
"segment_latency": 3446700
},
{
"switch_id": "00:00:00:00:00:00:00:07",
"port_no": 15,
"seq_id": 1,
"segment_latency": 3446700
}
]
},
{
"latency_ns": 3482233,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", sw_01 is deactivated - OK
"actual_state": "FAILED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:07",
"port_no": 15,
"seq_id": 0,
"segment_latency": 3482233
},
{
"switch_id": "00:00:00:00:00:00:00:01",
"port_no": 2,
"seq_id": 1,
"segment_latency": 3482233
}
]
},
{
"latency_ns": 3680203,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:07",
"port_no": 51,
"seq_id": 0,
"segment_latency": 3680203
},
{
"switch_id": "00:00:00:00:00:00:00:08",
"port_no": 47,
"seq_id": 1,
"segment_latency": 3680203
}
]
},
{
"latency_ns": 3370558,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:07",
"port_no": 49,
"seq_id": 0,
"segment_latency": 3370558
},
{
"switch_id": "00:00:00:00:00:00:00:09",
"port_no": 48,
"seq_id": 1,
"segment_latency": 3370558
}
]
},
{
"latency_ns": 4319796,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:08",
"port_no": 47,
"seq_id": 0,
"segment_latency": 4319796
},
{
"switch_id": "00:00:00:00:00:00:00:07",
"port_no": 51,
"seq_id": 1,
"segment_latency": 4319796
}
]
},
{
"latency_ns": 4081218,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:09",
"port_no": 48,
"seq_id": 0,
"segment_latency": 4081218
},
{
"switch_id": "00:00:00:00:00:00:00:07",
"port_no": 49,
"seq_id": 1,
"segment_latency": 4081218
}
]
},
{
"latency_ns": 3228426,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", sw_02 is deactivated - OK
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 6,
"seq_id": 0,
"segment_latency": 3228426
},
{
"switch_id": "00:00:00:00:00:00:00:07",
"port_no": 19,
"seq_id": 1,
"segment_latency": 3228426
}
]
},
{
"latency_ns": 3370558,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", sw_02 is deactivated - OK
"actual_state": "FAILED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:07",
"port_no": 19,
"seq_id": 0,
"segment_latency": 3370558
},
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 6,
"seq_id": 1,
"segment_latency": 3370558
}
]
},
{
"latency_ns": 4332,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", sw_02 is deactivated - OK
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 5,
"seq_id": 0,
"segment_latency": 4332
},
{
"switch_id": "00:00:00:00:00:00:00:08",
"port_no": 1,
"seq_id": 1,
"segment_latency": 4332
}
]
},
{
"latency_ns": 3832487,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:03",
"port_no": 5,
"seq_id": 0,
"segment_latency": 3832487
},
{
"switch_id": "00:00:00:00:00:00:00:09",
"port_no": 32,
"seq_id": 1,
"segment_latency": 3832487
}
]
},
{
"latency_ns": 4938,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "FAILED", sw_02 is deactivated - OK
"actual_state": "FAILED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:08",
"port_no": 1,
"seq_id": 0,
"segment_latency": 4938
},
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 5,
"seq_id": 1,
"segment_latency": 4938
}
]
},
{
"latency_ns": 3157360,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED",
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:09",
"port_no": 32,
"seq_id": 0,
"segment_latency": 3157360
},
{
"switch_id": "00:00:00:00:00:00:00:03",
"port_no": 5,
"seq_id": 1,
"segment_latency": 3157360
}
]
},
{
"latency_ns": 3487309,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED", - NOT OK as both switches are DEACTIVATED!!!
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:01",
"port_no": 1,
"seq_id": 0,
"segment_latency": 3487309
},
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 4,
"seq_id": 1,
"segment_latency": 3487309
}
]
},
{
"latency_ns": 3461928,
"speed": 10000000,
"available_bandwidth": 10000000,
"max_bandwidth": 10000000,
"default_max_bandwidth": 10000000,
"state": "DISCOVERED", - NOT OK as both switches are DEACTIVATED!!!
"actual_state": "DISCOVERED",
"round_trip_status": "FAILED",
"cost": 700,
"under_maintenance": false,
"enable_bfd": false,
"path": [
{
"switch_id": "00:00:00:00:00:00:00:02",
"port_no": 4,
"seq_id": 0,
"segment_latency": 3461928
},
{
"switch_id": "00:00:00:00:00:00:00:01",
"port_no": 1,
"seq_id": 1,
"segment_latency": 3461928
}
]
}
]
Expected result: both switches are deactivated and appropriate ISLs are FAILED.
To reproduce this case, the test System's able to fail an ISL if switches on both ends go offline during restart of network topology can be used.
The investigation of the calling Uni-ISL SETUP request is required.
yuliiamir commented
Minor: need to update test due to the comment https://github.com/telstra/open-kilda/pull/5507/files#r1415444053 when issue is fixed and test can be switched on.