Fix Session Timeout Issue During Install Step in Upgrade Script

[cdot65]

We have identified a bug in the upgrade script where a session timeout error can occur during the install step. Despite this error, the upgrade process continues on the remote device, but the script incorrectly reports a failure and halts execution.

Current Behavior:

During the upgrade process, if a session timeout occurs, the script outputs an error message and prematurely stops, as illustrated below:

🚀 katy-fw1: Attempting upgrade to version 10.1.4-h4 (Attempt 1 of 3).
Device 007054000242051 installing version: 10.1.4-h4
❌ katy-fw1: Upgrade error: Session timed out
🛑 katy-fw1: Critical error during upgrade. Halting script.

Expected Behavior:

The script should handle session timeouts more gracefully. Instead of halting the upgrade process, it should allow the upgrade to continue in the background and periodically check the progress through subsequent API calls for the job status.

Proposed Solution:

• Implement a mechanism within the script to detect session timeouts and differentiate them from actual upgrade failures.
• Upon detecting a session timeout, the script should log a warning instead of an error and continue to monitor the upgrade progress via API calls to the remote device.
• Provide clear logging to inform the user that the upgrade is proceeding in the background and include instructions on how to manually check the upgrade status if needed.

This enhancement will improve the reliability and user experience of the upgrade script, ensuring that upgrades can proceed smoothly even in the face of transient network issues or session timeouts.

Steps to Reproduce:

1. Initiate an upgrade using the script on a device with a known propensity for session timeouts.
2. Observe the script behavior when a session timeout occurs during the install step.

Acceptance Criteria:

• The script no longer halts on session timeouts during the upgrade install step.
• Users are informed about the continuation of the upgrade in the background and how to check its progress.
• The script successfully completes the upgrade process or provides a clear error message if the upgrade fails for reasons other than a session timeout.

[cdot65]

This issue appeared when upgrading ten virtual firewalls that are split between two hard disks on the same server. It may have been an I/O issue on the server but nonetheless should be considered as an opportunity to move away from pan-os-python's sync option for the software install method.