v0.8.0 Only selectively generates lcov reports

Question

v0.8.0 Only selectively generates lcov reports

Closed this issue 9 years ago · 37 comments

For my project, blanket only generates the lcov.dat file about half of the time I run it. It looks like testem is closing the browser session before blanket has a chance to send test results back to the coverage server.

Anyone else seeing this?

Answer 1 · 2016-02-23T23:04:25.000Z

I'm seeing something fairly similar (not sure it's identical, so I can move my comments to a different issue if it turns out this is different).

Background:
Ember 2.3
Ember-CLI 2.3.0-beta.1

When running on the command line (not via browser) to mimic our CI server, I can properly run code coverage info and get results back when filtering my tests (if I run few enough tests):

ember t --filter=service

However, running a normal ember test results in this:

$ ember t

<much snippage>

ok 989 PhantomJS 1.9 - JSHint - unit/transforms: unit/transforms/utc-test.js should pass jshint
Error: request aborted
    at IncomingMessage.onAborted (/home/ubuntu/ember-client/node_modules/ember-cli-blanket/node_modules/body-parser/node_modules/raw-body/index.js:269:10)
    at IncomingMessage.emit (events.js:104:17)
    at abortIncoming (_http_server.js:279:11)
    at Socket.socketOnEnd (_http_server.js:393:7)
    at Socket.emit (events.js:129:20)
    at _stream_readable.js:908:16
    at process._tickCallback (node.js:355:11)
Error: request aborted
    at IncomingMessage.onAborted (/home/ubuntu/ember-client/node_modules/ember-cli-blanket/node_modules/body-parser/node_modules/raw-body/index.js:269:10)
    at IncomingMessage.emit (events.js:104:17)
    at abortIncoming (_http_server.js:279:11)
    at Socket.socketOnEnd (_http_server.js:393:7)
    at Socket.emit (events.js:129:20)
    at _stream_readable.js:908:16
    at process._tickCallback (node.js:355:11)
Error: request aborted
    at IncomingMessage.onAborted (/home/ubuntu/ember-client/node_modules/ember-cli-blanket/node_modules/body-parser/node_modules/raw-body/index.js:269:10)
    at IncomingMessage.emit (events.js:104:17)
    at abortIncoming (_http_server.js:279:11)
    at Socket.socketOnEnd (_http_server.js:393:7)
    at Socket.emit (events.js:129:20)
    at _stream_readable.js:908:16
    at process._tickCallback (node.js:355:11)

Update:

Unsure of how to read this (possibly related to whether a container exists behind the scenes?). If I run just unit tests I'm seeing failures. But if I run acceptance tests and unit tests or integration tests and unit tests then things seem to work fine.

@arjansingh Does this lead to any more data on your end?

ember t --filter='/(acceptance|unit)/i'

Note: found out that filtering with regex worked here: qunitjs/qunit#906

Answer 2 · 2016-03-03T17:03:30.000Z

I have a similar issue - when I log the whole error https://github.com/sglanzer/ember-cli-blanket/blob/master/index.js#L14 rather than just the stack, I get:

{ [Error: request aborted]
  code: 'ECONNABORTED',
  expected: 660743,
  length: 660743,
  received: 524288,
  message: 'request aborted',
  status: 400,
  statusCode: 400,
  type: 'request.aborted' }

My guess is that when the coverage file gets above a certain size the express server is shut down before the coverage middleware (more specifically bodyParser), completes parsing the coverage POST request data. It probably happens intermittently because it's a race condition and it would only happen to your project when it get's above a certain size (our app has >2,500 tests).

Am investigating, but currently don't have any idea how to fix the issue.

Answer 3 · 2016-03-03T17:20:53.000Z

My guess is that when the coverage file gets above a certain size the express server is shut down before the coverage middleware (more specifically bodyParser), completes ..

Added some more logging and testem stops the express server after that error, so that theory is wrong

Answer 4 · 2016-03-04T13:58:25.000Z

I suspect that the 50mb limit set in our middleware is being superseded by another bodyParser in the middleware stack. are you using any other 'middleware' addons?

Answer 5 · 2016-03-04T16:48:41.000Z

No other middlewares (I verified by putting some logging in https://github.com/ember-cli/ember-cli/blob/8a5fc39aef990b32382f260c935a90e95d72359c/lib/tasks/test.js#L33)

I also logged the limit in bodyParser and it was set to 50mb

Answer 6 · 2016-03-04T16:51:33.000Z

@adamjmcgrath any chance you could run your tests in chrome and grab some stats from the network tab on /write-blanket-coverage - one thing I noticed today when debugging a qunit test (we use mocha here) is that it looked like we're posting multiple times

Answer 7 · 2016-03-04T16:59:48.000Z

We use mocha as well and there's only one POST to /write-blanket-coverage

Answer 8 · 2016-03-04T17:01:00.000Z

Does it show the post size? e.g. is size even an issue? or a red-herring?

On Mar 4, 2016, at 11:59 AM, Adam Mcgrath notifications@github.com wrote:

We use mocha as well and there's only one POST to /write-blanket-coverage

—
Reply to this email directly or view it on GitHub #112 (comment).

Answer 9 · 2016-03-04T17:08:08.000Z

The size is 2.4mb. The error from bodyParser is confusing:

expected: 660743,
length: 660743,
received: 524288,

Seems to suggest the file is ~660kb. I'll keep digging and let you know if I come up with anything

Answer 10 · 2016-03-04T17:09:51.000Z

FWIW, I was seeing three POSTs when I ran into this, I'll check back with more details when I work on that some more ...

Answer 11 · 2016-03-04T17:11:26.000Z

that is what I was seeing as well (not this size issue) - it looks like we’re listening for Qunit.done() - but that’s being called three times.

Not sure if something’s changed in qunit but it’s weird.

On Mar 4, 2016, at 12:09 PM, David Baker notifications@github.com wrote:

FWIW, I was seeing three POSTs when I ran into this, I'll check back with more details when I work on that some more ...

—
Reply to this email directly or view it on GitHub #112 (comment).

Answer 12 · 2016-03-08T17:09:45.000Z

So Qunit is unreliable? That would explain a lot regarding the intermittency. I'll also try and dig a little deeper in the next couple weeks.

Answer 13 · 2016-03-29T18:45:50.000Z

I just bumped into this issue as well trying to upgrade ember-cli and ember-blanket
I also see three requests aborted using qunit.

"ember-cli": "v2.4.3"
"ember-cli-blanket": "0.9.1",
"testem": "1.6.0"

Answer 14 · 2016-03-30T20:07:28.000Z

Looking at the logs I can see a couple of things:

If I run tests with a filter (~50 tests), coverage runs before tests.
If I run the full test suite (~2000 tests) coverage runs after tests and testem closes the session before the POST finishes.

Logs here:
https://gist.github.com/calderas/3374f43fde46f00005b2d2c45da78057

Answer 15 · 2016-03-31T20:00:49.000Z

@calderas et al, I'm looking at a change that reworks how we end the tests - will advise with a PR that you can try before we release

Answer 16 · 2016-04-01T13:15:40.000Z

@adamjmcgrath if you run ember test --server (which by definition will keep the server running) does it complete?

my thoughts last night didn't pan out - so adding browser side reporting for error conditions when the ajax call is fired is our best hope for improved diagnostics.

Answer 17 · 2016-04-01T14:25:58.000Z

ember test --server works fine for me.

Answer 18 · 2016-04-01T14:27:23.000Z

ok, cool - I’m prepping a PR now with some additional debug as well as the fix for uncovered/non imported modules.

On Apr 1, 2016, at 10:25 AM, calderas notifications@github.com wrote:

ember test --server works fine for me.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub #112 (comment)

Answer 19 · 2016-04-01T14:53:01.000Z

@adamjmcgrath the PR in #128 includes a new debug flag as well as some diagnostics about the post length, error handling etc. would be ideal if you could run against that branch and turn on the debugCLI:true flag in blanket-options

If that reveals some additonal info (or even gives a sense of how big the post is) that would be helpful.

Answer 20 · 2016-04-01T15:17:52.000Z

Sure I can run it and share output - will try and do it today, but it may be early next week.

Incidentally I fixed the issue I was having by changing the ajax request that posts the coverage to being synchronous (added the async: false option to https://github.com/sglanzer/ember-cli-blanket/blob/aa77828bfcebb9eed738c5589ef9ea01bb118652/vendor/start.js#L6)

This resolved the issue for me, my hunch was that if the request was synchronous the server wouldn't shut down until it was complete, which appeared to work. Apologies - I didn't share it with this thread because I thought my issue was separate to the one that qunit users where reporting here. Also, my company uses a fork of this project which has diverged quite a bit from the original.

Answer 21 · 2016-04-01T15:33:49.000Z

@adamjmcgrath - wow! that's super helpful. If I can get a reproducible case of it not writing to completion we can test async:false

would be interested in your fork

Answer 22 · 2016-04-01T17:25:26.000Z

Confirmed async:false does the trick !

Answer 23 · 2016-04-01T17:26:43.000Z

I’ll update my PR.

did you happen to see the debug line with size of posted data? curious to try and replicate.

On Apr 1, 2016, at 1:25 PM, calderas notifications@github.com wrote:

Confirmed async:false does the trick !

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub #112 (comment)

Answer 24 · 2016-04-01T19:58:41.000Z

Yes, this is what I get:

[ember-cli-blanket]:data size: 1475849
[ember-cli-blanket]:done writing coverage
[ember-cli-blanket]:data size: 1475849
[ember-cli-blanket]:done writing coverage
[ember-cli-blanket]:data size: 1475849
[ember-cli-blanket]:done writing coverage

Answer 25 · 2016-04-01T20:19:38.000Z

Thanks - that helps.

We need to add a guard against the triple write - that doesn’t happen with mocha AFAIK - so I’ll have to fire up a qunit example to debug that.

The PR #128 now includes async: false - I think it’s ready to go.

-jeff

On Apr 1, 2016, at 3:58 PM, calderas notifications@github.com wrote:

Yes, this is what I get:

[ember-cli-blanket]:data size: 1475849
[ember-cli-blanket]:done writing coverage
[ember-cli-blanket]:data size: 1475849
[ember-cli-blanket]:done writing coverage
[ember-cli-blanket]:data size: 1475849
[ember-cli-blanket]:done writing coverage
—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub #112 (comment)

Answer 26 · 2016-04-02T08:13:34.000Z

I finally dug into this tonight. The root cause, I believe, is testem, specifically: testem/testem#616

Basically on test completion, testem indiscriminately kills every open socket to its server, which explains why large coverage reports (which might still be transmitting) are often cut off mid transit.

I tested locally and also agree that the solution of using blocking Ajax calls (async: false) should do the trick. Transmitting the results synchronously effectively prevents the browser from declaring the test run done until the report is fully transmitted.

Great find @adamjmcgrath!

Answer 27 · 2016-04-05T18:55:09.000Z

Confirmed, #128 worked on our end as well. I believe this issue can be closed! 👍

Answer 28 · 2016-04-05T20:03:27.000Z

Well... Let's wait for the PR to be merged. 😊

On Tuesday, April 5, 2016, David Baker notifications@github.com wrote:

Confirmed, #128 #128
worked on our end as well. I believe this issue can be closed! [image:
👍]

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#112 (comment)

Arjan

Answer 29 · 2016-04-05T20:34:27.000Z

Believe it was 4 days ago ;-)

On Apr 5, 2016, at 2:05 PM, Arjan Singh notifications@github.com wrote:

Well... Let's wait for the PR to be merged. 😊

On Tuesday, April 5, 2016, David Baker notifications@github.com wrote:

Confirmed, #128 #128
worked on our end as well. I believe this issue can be closed! [image:
👍]

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#112 (comment)

Arjan
—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

Answer 30 · 2016-04-05T21:17:23.000Z

Let's wait until 0.9.2 is published on npm then? then? I just want to make sure this bad boy is live.

Answer 31 · 2016-04-05T21:22:19.000Z

Waiting on @sglanzer to cut a new build.

-jeff

On Apr 5, 2016, at 3:42 PM, David Baker notifications@github.com wrote:

Believe it was 4 days ago ;-)

On Apr 5, 2016, at 2:05 PM, Arjan Singh notifications@github.com wrote:

Well... Let's wait for the PR to be merged. 😊

On Tuesday, April 5, 2016, David Baker notifications@github.com wrote:

Confirmed, #128 #128
worked on our end as well. I believe this issue can be closed! [image:
👍]

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#112 (comment)

Arjan
—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

Answer 32 · 2016-04-06T05:15:07.000Z

Sorry, I'll handle that tomorrow - we need to figure out the NPM publishing permissions so that I'm not a bottleneck on this

Answer 33 · 2016-04-06T18:11:07.000Z

Published 0.9.2

Answer 34 · 2016-04-06T18:23:02.000Z

Thanks @sglanzer

On Apr 6, 2016, at 1:11 PM, Steven Glanzer notifications@github.com wrote:

Published 0.9.2

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub #112 (comment)

Answer 35 · 2016-04-06T23:24:37.000Z

👍

On Wednesday, April 6, 2016, Jeff Schilling notifications@github.com
wrote:

Thanks @sglanzer

On Apr 6, 2016, at 1:11 PM, Steven Glanzer <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Published 0.9.2

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub <
#112 (comment)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#112 (comment)

Arjan

Answer 36 · 2016-04-07T02:15:46.000Z

Any objections to me closing this (since I filed it)?

Answer 37 · 2016-04-07T02:17:57.000Z

Thanks. That would be awesome

-jeff

On Apr 6, 2016, at 9:15 PM, Arjan Singh notifications@github.com wrote:

Any objections to me closing this (since I filed it)?

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub