[fixed] xcpretty hangs with xcode 8 beta (8S128d)
mindows opened this issue · 29 comments
When I execute tests with xcodebuild test it exits with an exit code of 65 (some tests failed). However, when I pipe it through xcpretty it hangs indefinitely.
This is the last line I see in my console as it hangs forever
Executed 1801 tests, with 23 failures (3 unexpected) in 218.200 (219.161) seconds
I guess the latest xcodebuild (Beta) has changed how it exits the process? Has anyone else seen this issue?
thanks in advance
To manage this, we simply use the PIPESTATUS variable :
${XCODEBUILD_COMMAND} | xcpretty -s 2>&1
XCODEBUILD_RC=${PIPESTATUS[0]}
@kenji21 we use PIPESTATUS also, but the problem is that it hangs on the first line actually. Are you using the new xcode beta?
I filed a radar on this. It's clearly a bug in xcodebuild (xcode 8 beta 1). I also spoke to an apple engineer at WWDC, and he suspected that they forgot to close file descriptor(s) in xcodebuild.
I wrote a hacky python wrapper around xcodebuild to pipe every stdout & stderr and exit when I see the output lines I'm expecting. It's something like Test Suite \'(Selected|All) tests\' (passed|failed)
Hopefully, beta 2 or 3 will fix this issue.
@mindows nice one! Thanks for that : -)
(sidenote, do you also see the barrage of "The operation requires a selected developer portal team." errors in stderr? I've filed a bug for it, but I wonder if there's something wrong with my config)
Thanks for reporting! I'll take a look into it.
Instead of xcodebuild test I use the new command-line options xcodebuild build-for-testing and xcodebuild test-without-building. Works like a charm. ;)
fyi: build-for-testing appears to terminate fine but test-without-building still hangs in our explorations (google/arc-xcode-test-engine#7).
+sharing the python script would be rad :)
Okay, so test-without-building worked on a real device for me. Using the simulator I noticed that the simulator was started on the CI machine including the UI and remained running after the tests finished. Once I manually terminated the simulator, the test-without-building | xcpretty finished with return code 0. It seems, that the simulator is not terminated and keeps the xcodebuild process alive.
FWIW I'm still seeing this in Xcode 8 Beta 3 (8S174q)
It seems like Apple has fixed the issue with today's beta, could you check if that works for you?
It's working for me now with beta 4.
closing, please reopen if you still see the issue in Xcode 8
@supermarin i'm seeing this happen on my tests with Xcode 8 (installed from AppStore) when our project's tests have a failure. If they all pass, this doesn't seem to happen.
Here's a sample command I'm using:
xcodebuild -workspace Sample.xcworkspace -scheme 'Sample Target' test-without-building -destination 'platform=iOS Simulator,name=iPhone 6s Plus,OS=10.0' | xcpretty
Is anyone else still getting this to happen in Xcode 8?
@proskd which version of xcpretty are you running?
xcpretty --version
I was running 0.1.11.
Before updating, I noticed that without changing anything, one or two runs did complete successfully (that is, they exited the process after test failures).
I then updated to 0.2.4 and got it to hang again with the same command.
Even if I pass the set -o pipefail && before hand (as suggested in the readme.md, or use the other suggested workaround) it still will hang.
EDIT: it seems if i simply wait long enough, xcpretty will actually end itself? maybe i'm just not waiting long enough?
have you tried NSUnbufferedIO=YES xcodebuild ...?
That didn't seem to help, but I'd say at this point as my edit above indicated, I think if I wait long enough (several minutes after seeing ** TEST EXECUTE FAILED ** in my console), eventually xcpretty does output a bunch of stuff and exits. I'm just not really clear what it's doing during this period where nothing is being output to the console...
I think the problem sounds like Xcode not closing the file handle properly.
Just to test one more case, did you try running it with tee in the middle?
xcodebuild ... | tee xcodebuild.log | xcpretty
And which exact Xcode 8 version?
Version 8.0 (8A218a)
I wasn't using tee, but i did just try that now per your suggestion and it doesn't appear to make a difference one way or the other.
mind posting the raw xcodebuild.log?
Sure, here you go!
@proskd I'm getting the same issue with the newest xcpretty and xcode version. did you solve it?
Thanks!
@noa-houzz unfortunately no, I never did. I ended up giving up since I had other things I needed to do, but the end result was that we couldn't use xcpretty due to the hangs.
One thing I can tell you though is that unrelated to xcpretty, we did have an issue where our unit test target would occasionally (and seemingly randomly) hang, due to using the app as the execution container of the tests (which is normally the "default" in Xcode). We tracked this issue down to the tests and apps competing for things they shouldn't have been competing for (in this case, CoreData objects) so we were able to work around that.
It's possible the above issue was the underlying root cause, but I never actually validated it. Thanks for the friendly reminder, I'll have to go back and check.