Add Support for Safaridriver Debug Logs
lread opened this issue · 4 comments
Problem/Opportunity
Safaridriver is different from other WebDriver implementations.
It has fewer features, and its docs induce head-scratching, at least for me.
When diagnosing mysterious behavior, it can be very helpful to enable WebDriver logging.
We support this easily for all WebDriver implementations, except for safaridriver, via :log-stdout and :log-stderr (and :driver-log-level).
When I need safaridriver logging, I will hack some temporary code into Etaoin to get it to spit out safaridriver debug logs. After the mystery is solved, the hack is abandoned, and how I achieved it fades from my memory.
Technical Details
Safaridriver debug logging is enabled via its --diagnose command line option.
There don't seem to be any logging levels to set.
Other WebDrivers send output to stdout/stderr, but safaridriver always writes its logs to ~/Library/Logs/com.apple.WebDriver/safaridriver.{safaridriver pid}.{random chars}.txt.
Proposed Solution
It would be nice to have a way to ask Etaoin to spit out safaridriver logs.
Interpret :driver-log-level of debug to turn safaridriver logging on.
Stream to log to stdout; this will allow the reuse of the existing :log-stdout feature.
Alternative Solutions
Don't stream log; spit it to stdout when the safaridriver process is closed.
This would be simpler to implement but less useful.
Additional context
The safaridriver PID is part of its log filename. Determining the safaridriver PID should be straightforward in jdk9+, but we currently still support jdk8+. If I decide to stick with jdk8+ support, whatever technique I come up with only has to work on macOS, the only platform where safaridriver is currently supported.
Action
I'll likely tackle this one. If he remembers, future-me might thank me.
Although safaridriver --help suggests reading its man page for details:
Usage: safaridriver [options]
-h, --help Prints out this usage information.
--version Prints out version information and exits.
-p, --port Port number the driver should use. If the server
is already running, the port cannot be changed.
If port 0 is specified, a default port will be used.
--enable Applies configuration changes so that subsequent WebDriver
sessions will run without further authentication.
--diagnose Causes safaridriver to log diagnostic information for
all sessions hosted by this instance. See the safaridriver(1)
man page for more details about diagnostic logging.
... man safaridriver does not find a man page, at lest on macOS 14.3.1.
The only recent-ish manpage I found on the web is here: https://gist.github.com/foolip/a0d7ac345a67a09a709ef7b28b3924ce
Contents pasted just in case gist goes poof:
SAFARIDRIVER(1) BSD General Commands Manual SAFARIDRIVER(1)
NAME
safaridriver -- Safari WebDriver REST API service
SYNOPSIS
safaridriver -p port [-h | --help] [--version] [--enable] [--diagnose]
DESCRIPTION
The safaridriver utility is used to launch an HTTP server that implements the Selenium WebDriver
REST API. When launched, safaridriver allows for automated testing of web content using the ver-
sion of Safari that is installed with macOS.
safaridriver supports several capabilities that can customize an automation session's behavior for
a particular testing purpose. Capabilities are provided as arguments when requesting a new ses-
sion. The capabilities supported by safaridriver are listed in the OPTIONS section below. Unless
noted below, the values of requested capability keys are not read and are assumed to be true if
present, and false otherwise.
To use capabilities in your tests, please refer to the relevant 3rd-party documentation to learn
how to request extra capabilities with the WebDriver client library that you are using.
OPTIONS
Command line options
-p, --port <port>
Specifies the port on which the HTTP server should listen for incoming connections. If
the port is already in use or otherwise unavailable, safaridriver will exit immediately
with a non-zero return code.
-h, --help
Prints a usage message and exits.
--version
Prints version information and exits.
--enable
Applies configuration changes so that subsequent WebDriver sessions will run without fur-
ther authentication. This includes checking "Enable Remote Automation" in Safari's
Develop menu. The user must authenticate via password for the changes to be applied.
When this option is specified, safaridriver exits immediately without starting up the
REST API service. If the changes were successful or already applied, safaridriver exits
0; otherwise, safaridriver exits >0 and prints an error message to stderr.
--diagnose
Enables diagnostic logging for all sessions hosted by this safaridriver instance. See
DIAGNOSTICS for more information.
Session Creation Capabilities
browserName
safaridriver can only create WebDriver sessions for Safari. If the value of browserName
is not equal to `Safari', session creation will fail.
Values of browserName are compared case-insensitively.
browserVersion
safaridriver will only create a session using hosts whose Safari version matches the
value of browserVersion.
Browser version numbers are prefix-matched. For example, if the value of browserVersion
is `12', this will allow hosts with a Safari version of `12.0.1' or `12.1`.
platformName
If the value of platformName is `mac' or `macOS', safaridriver will only create a session
using the macOS host on which safaridriver is running.
If the value of platformName is `iOS', safaridriver will only create a session on a
paired iOS device or simulator.
Values of platformName are compared case-insensitively.
safari:platformVersion
safaridriver will only create a session using hosts whose OS version matches the value of
safari:platformVersion.
OS version numbers are prefix-matched. For example, if the value of
safari:platformVersion is `12', this will allow hosts with an OS version of `12.0' or
`12.1' but not `10.12'.
safari:platformBuildVersion
safaridriver will only create a session using hosts whose OS build version matches the
value of safari:platformBuildVersion. example of a macOS build version is `18E193'.
On macOS, the OS build version can be determined by running the sw_vers(1) utility.
safari:useSimulator
If the value of safari:useSimulator is true, safaridriver will only use iOS Simulator
hosts.
If the value of safari:useSimulator is false, safaridriver will not use iOS Simulator
hosts.
NOTE: An Xcode installation is required in order to run WebDriver tests on iOS Simulator
hosts.
safari:deviceType
If the value of safari:deviceType is `iPhone', safaridriver will only create a session
using an iPhone device or iPhone simulator.
If the value of safari:deviceType is `iPad', safaridriver will only create a session
using an iPad device or iPad simulator.
Values of safari:deviceType are compared case-insensitively.
safari:deviceName
safaridriver will only create a session using hosts whose device name matches the value
of safari:deviceName. Device names are compared case-insensitively.
NOTE: Device names for connected devices are shown in iTunes. If Xcode is installed,
device names for connected devices are available via the output of instruments(1) and in
the Devices and Simulators window (accessed in Xcode via "Window > Devices and Simula-
tors").
safari:deviceUDID
safaridriver will only create a session using hosts whose device UDID matches the value
of safari:deviceUDID. Device UDIDs are compared case-insensitively.
NOTE: If Xcode is installed, UDIDs for connected devices are available via the output of
instruments(1) and in the Devices and Simulators window (accessed in Xcode via "Window >
Devices and Simulators").
Other Capabilities
safari:automaticInspection
This capability instructs Safari to preload the Web Inspector and JavaScript debugger in
the background prior to returning a newly-created window. To pause the test's execution
in JavaScript and bring up Web Inspector's Debugger tab, you can simply evaluate a
debugger; statement in the test page.
safari:automaticProfiling
This capability instructs Safari to preload the Web Inspector and start a Timeline
recording in the background prior to returning a newly-created window. To view the
recording, open the Web Inspector through Safari's Develop menu.
safari:diagnose
This capability requests that diagnostics be enabled for the session. See DIAGNOSTICS
for more information.
webkit:WebRTC
This capability allows a test to temporarily change Safari's policies for WebRTC and
Media Capture. The value of the webkit:WebRTC capability is a dictionary with the fol-
lowing sub-keys, all of which are optional:
DisableInsecureMediaCapture
Normally, Safari refuses to allow media capture over insecure connections. This
capability suppresses that restriction for testing purposes. For example, it would
allow a test to exercise media capture code paths using a local test web server
that is not configured to use HTTPS.
DisableICECandidateFiltering
To protect a user's privacy, Safari normally filters out WebRTC ICE candidates that
correspond to internal network addresses when capture devices are not in use. This
capability suppresses ICE candidate filtering so that both internal and external
network addresses are always sent as ICE candidates.
EXIT STATUS
The safaridriver utility exits 0 on success, and >0 if an error occurs.
ERRORS
When a REST API command fails, safaridriver includes a detailed error message in the response. If
you use a 3rd-party library on top of the REST API service, consult the library's documentation
for how to access these error messages.
SESSION CREATION
safaridriver can create WebDriver sessions using Safari on a macOS machine, a paired iOS device,
or an iOS simulator. Capabilities listed in the Session Creation Capabilities subsection provide
criteria that affect which hosts safaridriver will attempt to use when handling a New Session com-
mand. A host must match all criteria to be usable. If no hosts match all of the criteria, then
the New Session command will fail. If multiple hosts match all of the criteria, the order in
which safaridriver will use them is unspecified, except that booted simulators are used before
unbooted simulators.
If a simulator host matches the criteria but is not booted, safaridriver will attempt to boot the
simulator instance and wait for it to become usable. If Safari is not running on a host that oth-
erwise matches the criteria, safaridriver will automatically launch Safari and wait for it to
become usable. If a host's Safari instance is associated with an inactive WebDriver session,
safaridriver will replace the old session unless the session was manually interrupted by the user
or is being inspected by Web Inspector.
NOTES
For security reasons, the HTTP server accepts connections from localhost only. The HTTP server can
accept connections from multiple test clients. Safari on macOS and iOS can only host one WebDriver
session at a time, so it is not recommended to run multiple safaridriver instances at the same
time.
safaridriver is typically executed manually at the command line or automatically by a WebDriver
client library. The Selenium project provides client libraries for most popular programming lan-
guages. More information is available on the Selenium project website:
https://www.seleniumhq.org/
SEE ALSO
A summary of which REST API endpoints safaridriver supports, as well as the Safari version in
which each endpoint became available, can be found on the Apple Developer website:
https://developer.apple.com/
STANDARDS
safaridriver implements the W3C WebDriver specification:
https://www.w3.org/TR/webdriver/
DIAGNOSTICS
When filing a bug report against safaridriver, it is highly recommended that you capture and
include diagnostics generated by safaridriver. This can be accomplished in several ways:
To diagnose a single session, pass the safari:diagnose capability when requesting a new session.
To diagnose all sessions from one safaridriver instance, use the --diagnose command line option.
To diagnose all sessions in all instances of safaridriver, set the DiagnosticsEnabled default in
the com.apple.WebDriver domain to YES using defaults(1).
Diagnostic files are saved to ~/Library/Logs/com.apple.WebDriver/ and are uniquely named using the
pid of safaridriver and a timestamp. When using the safari:diagnose capability to turn on diag-
nostics for a particular session, diagnostic files additionally include the session identifier in
file names.
Darwin November 2, 2019 Darwin
I'll stick with my plan to use the --diagnose option.
In testing, I have discovered that for ~/Library/Logs/com.apple.WebDriver/safaridriver.{safaridriver pid}.{random chars}.txt, the {safaridriver pid} does not match the safaridriver process pid:
Clear out any existing logs...
rm ~/Library/Logs/com.apple.WebDriver/*
Start safaridriver:
$ safaridriver --diagnose --port 9999
From a separate terminal...
Force some logging:
$ curl http://localhost:9999/status
{"value":{"message":"","ready":true}}
Have a look at the generated log filename:
$ ls -l ~/Library/Logs/com.apple.WebDriver
total 8
-rw-r--r--@ 1 lee staff 120 19 Mar 12:11 safaridriver.50629.qzqwjh.txt
Look at pid for safaridriver process.
$ ps -ef
UID PID PPID C STIME TTY TIME CMD
...
502 50628 13512 0 12:10pm ttys000 0:00.02 safaridriver --diagnose --port 9999
...
Zut alors, 50628 does not match the filename PID of 50629.
So what process has a PID of 50629?
$ ps -ef
UID PID PPID C STIME TTY TIME CMD
...
502 50629 1 0 12:10pm ?? 0:00.14 /System/Library/PrivateFrameworks/WebDriver.framework/Versions/A/XPCServices/com.apple.WebDriver.HTTPService.xpc/Contents/MacOS/com.apple.WebDriver.HTTPService
...
And I don't see a process connection between the com.apple.WebDriver.HTTPService process and the safaridriver process. (Ie. safaridriver is not the parent process of com.apple.WebDriver.HTTPService).
Huh. That all jives with safaridriver being user-cruel.
Option 1 - assume only 1 safaridriver running
I could make the assumption that the doc recommendations of only having one safaridriver running are followed. Could check for new log files created after launching safaridriver. There should only be one. Assume that the new log file belongs to the safaridriver process.
Pro:
- simple
Con:
- breaks if someone decides to run multiple safaridriver processes at the same time
Option 2 - dummy correlation id
Call WebDriver status with a dummy parameter set to a unique id. This unique id would be written to the log file and be used to correlate the file to the process.
Let's see if that might work:
# clear out existing logs again..
rm ~/Library/Logs/com.apple.WebDriver/*
# start up safaridriver
safaridriver --diagnose --port 9999
From another terminal:
$ curl "http://localhost:9999/status?etaoin-dummy=3b990e7a-64a0-4d3f-ad40-e0fbb22c5e5e"
{"value":{"message":"","ready":true}}
And let's take a peek to see if that dummy value is written:
$ ls ~/Library/Logs/com.apple.WebDriver
safaridriver.51172.rqvera.txt
$ cat
2024/03/19 17:37:47:615 HTTPServer: GET http://localhost:9999/status?etaoin-dummy=3b990e7a-64a0-4d3f-ad40-e0fbb22c5e5e
2024/03/19 17:37:47:615 HTTPServer: --> HTTP 200
Pro:
- absolutely identifies log file to safaridriver
Con:
- breaks if safaridriver stops logging request, probably unlikely?
- breaks if safaridriver fails status request because of unrecognized parameter
Option 3 - Use lsof
Safaridriver is running on port 9999 (See setup in option 2)
$ lsof -ti :9999
51172
And this is the correct pid for the log file.
Pro:
- easy peasy
Con:
- breaks if safaridriver fixes what seems to be a bug, the pid should be for safaridriver process, not the com.apple.WebDriver.HTTPService process right?
I am looking at optionally pretty printing log output as part of this work. My initial idea was to show prefixes like:
safari 1234 out| some stdout log line
safari 1234 err| some stderr log line
safari 1234 log| some safaridriver logfile log line
...
Where safari is the web driver type, 1234 is the web driver port, and out, err, and log distinguish the log line source.
A few rabbit holes later, I have not found a way to preserve the log line order of stdout and stderr and also distinguish them. So when pretty printing, my next plan is to redirect stderr to stdout and lose the distinction.
safari 1234| some stdout log line
safari 1234| some stderr log line
safari 1234 log| some safaridriver logfile log line
...
I think, for this use case, distinguishing log lines will work out ok, but we'll see.