Is there any timeout for WinAppDriver listening on an IP and port?. Also any info on WinAppDriver status 13.
RameshR66 opened this issue · 9 comments
Hello,
We have automated our appium testcases using WinAppDriver. It is running in remote machine through task scheduler and is scheduled to run in every 30 minutes every day. It runs for a day and then WinAppDriver stops listening in on the specified IP and port. When we checked the WinAppDriver, it is showing status as mentioned below -
{"status":13," value":{"error":"unknown error","message":"An unknown error occurred in the remote end while processing the command."}}
During this time, the concerned task parameters shows as follows -
- Last Run Time - the time during which WinAppDriver stopped listening.
- Next Run Time - the current time the task is scheduled to run next.
You can find almost 12 hours of difference between these two parameter timings. This means that the task is trying to run as scheduled, but it is unable to do so as WinAppDriver is not listening to the mentioned IP and port anymore.
Once we restart the WinAppDriver it is working, and the task runs fine.
Please let us know if there is a default timeout in WinAppDriver to listen in on an IP and Port. Also, it would be great if we could have any information about the WinAppDriver status 13.
Thanks.
@RameshR66 can you give some insight as to how these tests are ran? Are you closing and restarting the WinAppDriver session in between test cases? How many test cases are you executing?
I will occasionally see this error in a suite of around 550 UI tests. Usually, it will impact maybe one test case but by the time the tear down is called and the next test case creates a new session, all subsequent test cases will not be impacted.
@Tree55Topz
The tests are run through a task in task scheduler. WinAppDriver doesn't really support unattended mode of running testcases on GUI based applications, so we did workarounds on it.
For Example -
We made sure WinAppDriver is working when user either minimizes or disconnects the remote desktop connection.
The main idea is to maintain the user session in the VM.
Are you closing and restarting the WinAppDriver session in between test cases? - No, we restart only when we notice this error. In our case the subsequent test cases fail as well, and the scheduled task doesn't run as mentioned in the initial discussion.
How many test cases are you executing? - At the moment, we are only running around 20 testcases that are solely meant to check WinAppDriver. These testcases are run every 30 minutes every day to check if the driver works in unattended mode properly. [With the work arounds that we have implemented]
What we noticed so far -
The task itself doesn't run, and we think the issue might be to do with initializing a session with the WinAppDriver. We placed few more log files to get more clarity on this.
Please let us know if you need further information to get better understanding and it would be really great if you could let us know your thoughts on this.
@RameshR66 its usually good practice to have your test cases StartUp and TearDown for each independent scenario. Usually, test classes should inherit some sort of TestBaseClass where the StartUp and TearDown logic is defined.
StartUp usually should launch a new winappdriver session, start the app and things like that. TearDown is ran automatically after a test class finishes and should define the logic for cleaning up the session. I use NUnit as my test harness, but I am sure whatever test runner you are using would probably use something similar.
There are tons of examples online about how to structure your StartUp and TearDown methods, I would consider looking into that. This way, if this error fails one test you at least will have a fresh session going for your next one. Then, determined by how often you see this error you may want to start debugging that specific port to see what might going on. I will say I have seen this error but it has been in rare cases and hasnt been worth the time to try to throw some additional Try/Catch logic to handle it.
And your test cases are solely meant to check WinAppDriver? Not sure what that means.. You are testing an app correct?
@RameshR66 also keep in mind this very well could be due to how your test VM is set up. Did you ensure that you configured your registry settings on the host to allow for minimized RDC connections to share the GUI? You need to always have at least some host with these settings. If you are disconnecting from the VM and there is nothing hosting it, WinAppDriver will not be able to run properly
@Tree55Topz
I misunderstood this question "Are you closing and restarting the WinAppDriver session in between test cases?" My apologies!.
Like you mentioned, we have setup StartUp and TearDown logic for each test case. We are using "Unit Test Project (.Net Framework)" instead of "NUnit" as it will provide an .exe file to run directly in my local to test instead of referencing to a console through .dll file.
Each test case is run through separate session with WinAppDriver as per our logic. The problem is once the error occurs, the subsequent test cases fails as well even if it is run through different sessions. When this happens, we login to remote desktop and restart the WinAppDriver.exe manually.
You are testing an app correct? - Nah! I am not in WinAppDriver Dev\QA team :)
Did you ensure that you configured your registry settings on the host to allow for minimized RDC connections to share the GUI?- Yes we did.
If you are disconnecting from the VM and there is nothing hosting it, WinAppDriver will not be able to run properly - This is the case here. We did workarounds to keep the VM with active session to run WinAppDriver. But its not running as expected. It runs for a while and errors out. :(
Our requirement is to run the GUI based test cases in unattended mode. We don't want any user interaction to interrupt the scheduled tasks. We are also looking into "Appium Server GUI" for this which is a cross platform utility which internally uses WinAppDriver for windows applications automation.
@RameshR66
Are You using anything like AzureDevOps pipelines/Jenkins pipelines/GitHub Actions?
What I understand here is that You are just opening WinAppDriver console once and then run suite of tests every 30 minutes.
Maybe better approach is to wrap it up in some pipeline, open WinAppDriver before starting tests and then close WinAppDriver after executing it and use built-in scheduler in pipelines?
In my current project we have ad-hoc scheduler (when new changes are pushed to main) and mindnight scheduler.
Every single time the task is started, VM is started, WinAppDriver console is started, Tests are executed, WinAppDriver is closed and VM is stopped to avoid bigger costs.
@kamilchodola
Are You using anything like AzureDevOps pipelines/Jenkins pipelines/GitHub Actions? - Yes, we do use AzureDevOps pipelines to update the test cases to automate.
Can you please elaborate on your scheduler activities?
As per my understanding, you are internally through code open a VM session using a credential and invoke the scheduler in there to run and close the same. Is this correct?
@RameshR66
Let me describe how we run our tests:
1st stage - run VM using webhook to Azure Portal
2nd stage - checkout code, start WinAppDriver console, build code, run tests, stop WinAppDriver, cleanup of env
3rd stage - stop VM
1 and 3 are ensuring we spent only minimum amount of $$$ for best possible VMs to run tests as quickly as possible
2 is establishing environment, testing and cleaning up
By this approach we are sure that every single time everything is fresh (version of application for test and version of tests) + every component is opened again. There is no dependencies to old runs (since VM was restarted) and I can say it is working really stable and entire configuration did not take more than 5 minutes.
@kamilchodola
I am new to WebHook and am not familiar with the concept. My understanding is that whenever you have a new pull request for new test cases, you approve and run it through the pipeline. While this is running, you establish connection to the VM and perform various stages as mentioned above.
If so, the test suite is run only when we add new test cases. In our case, we should be able to run and test the test suite whenever we want independently, and it should be automated and scheduled to run every 30 minutes or an hour depending on the amount of test cases.
We achieved this, but sometimes WinAppDriver throws the mentioned error above and literally unable to communicate after that. The subsequent test cases fail as well.