Keep service tests green 💚

Question

Keep service tests green 💚

paulmelnikow opened this issue 7 years ago · 16 comments

We want our service tests to hit the real services. This makes some of them unreliable – out of necessity, not sloppiness. There's a long discussion of the test strategy in #927.

I’d like to use this issue as a place to identify and fix flaky service tests.

IcedFrisby/IcedFrisby#71 will allow us to set a per-test timeout() and per-test retry(), which should allow us to keep flaky tests green most of the time.

Of course we want the requests to be fast. Right now they fail after 2 seconds, which is a fairly long time to be waiting for a badge. Since readme images are served through GitHub’s proxy (camp) which has its own timeout, perhaps we should choose a threshold in relation to that. IcedFrisby has a separate method .expectMaxResponseTime() which we could use to assert the response time. We could also generate a tidy report of slow badges and print it as part of our daily run. An often-red build is not the best way to accomplish this! 😄

Answer 1 · 2017-12-08T16:48:09.000Z

This is a list of failures from a recent test run on master:

bower
codeclimate: let's track in #1329
suggest (This is not even a service! Maybe it broke in #1282?)

  1) Bower
       licence

  [ GET http://localhost:1111/bower/l/bootstrap.json ]:
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.


  2) Bower
       custom label for licence

  [ GET http://localhost:1111/bower/l/bootstrap.json?label="my licence" ]:

      AssertionError: expected { name: '"my licence"', value: 'MIT' } to deeply equal { name: 'my licence', value: 'MIT' }
      + expected - actual

       {
      -  "name": "\"my licence\""
      +  "name": "my licence"
         "value": "MIT"
       }

      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:142:38)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  3) Bower
       custom label for version

  [ GET http://localhost:1111/bower/v/bootstrap.json?label="my version" ]:
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.


  4) Bower
       custom label for pre version

  [ GET http://localhost:1111/bower/vpre/bootstrap.json?label="pre version" ]:
     ValidationError: child "name" fails because ["name" must be one of [pre version]]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:663:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:299:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:738:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  5) Bower
       Pre Version for Invaild Package

  [ GET http://localhost:1111/bower/vpre/it-is-a-invalid-package-should-error.json ]:
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.


  6) Bower
       licence for Invaild Package

  [ GET http://localhost:1111/bower/l/it-is-a-invalid-package-should-error.json ]:
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.


  7) Code Climate
       test coverage score

  [ GET http://localhost:1111/codeclimate/c/Nickersoft/dql.json ]:
     ValidationError: child "value" fails because ["value" must be one of [A, B, C, D, F, unknown]]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:663:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:299:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:738:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  8) suggest
       issues, forks, stars and twitter

  [ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     TypeError: Expected an Array in the path 'badges.?' but got undefined
      at assertImpl (node_modules/check-types/src/check-types.js:730:11)
      at assertPredicate (node_modules/check-types/src/check-types.js:718:5)
      at Function.array (node_modules/check-types/src/check-types.js:710:14)
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:122:23)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  9) suggest
       license

  [ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     TypeError: Expected an Array in the path 'badges.?' but got undefined
      at assertImpl (node_modules/check-types/src/check-types.js:730:11)
      at assertPredicate (node_modules/check-types/src/check-types.js:718:5)
      at Function.array (node_modules/check-types/src/check-types.js:710:14)
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:122:23)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  10) suggest
       license for non-existing project

  [ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     TypeError: Expected an Array in the path 'badges.?' but got undefined
      at assertImpl (node_modules/check-types/src/check-types.js:730:11)
      at assertPredicate (node_modules/check-types/src/check-types.js:718:5)
      at Function.array (node_modules/check-types/src/check-types.js:710:14)
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:122:23)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  11) suggest
       license when json response is invalid

  [ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     TypeError: Expected an Array in the path 'badges.?' but got undefined
      at assertImpl (node_modules/check-types/src/check-types.js:730:11)
      at assertPredicate (node_modules/check-types/src/check-types.js:718:5)
      at Function.array (node_modules/check-types/src/check-types.js:710:14)
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:122:23)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  12) suggest
       license when html_url not found in GitHub api response

  [ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     TypeError: Expected an Array in the path 'badges.?' but got undefined
      at assertImpl (node_modules/check-types/src/check-types.js:730:11)
      at assertPredicate (node_modules/check-types/src/check-types.js:718:5)
      at Function.array (node_modules/check-types/src/check-types.js:710:14)
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:122:23)
      at node_modules/icedfrisby/lib/icedfrisby.js:760:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1272:33)
      at node_modules/icedfrisby/lib/icedfrisby.js:1251:29
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:562:11)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1059:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

Answer 2 · 2017-12-22T04:05:20.000Z

Bower should be fixed in #1383. Thanks @RedSparr0w!

Answer 3 · 2017-12-22T04:09:43.000Z

Also Code Climate Fixed in #1368

Although if you look at our comments after the merge we definitely still need to fix some problems,
As the tests do not have 100% coverage for all of their badges, the other ones won't currently fail.

Answer 4 · 2017-12-26T05:06:55.000Z

Down to 13 errors

Github - Seems to be consistently timing out, but looks to be running correctly without issue for me.
Suggest - Not sure what this is, haven't properly looked into it yet.
PHP version from .travis.yml - Not sure why this is failing as it actually shows as expected.

Answer 5 · 2017-12-27T04:45:24.000Z

Suggest is for a web service that provides badge suggestions to the front end. You can try it out by pasting a github URL into the search box at the top, and clicking Suggest Badges.

Answer 6 · 2017-12-27T05:03:22.000Z

Link for reference: https://circleci.com/gh/badges/shields/643

Github hit counter for nonexistent repo is one that I see timing out really often. We may as well bump that one up. Possibly hit counter too. The others I haven't seen time out before.

PHP version - maybe it's a Travis rate limit?

Answer 7 · 2017-12-27T05:08:57.000Z

Opened #1398 for suggest.

Answer 8 · 2017-12-27T22:01:12.000Z

https://circleci.com/gh/badges/shields/728

A strange new error for GitHub open issues by multi-word label is > zero today:

[ GET http://localhost:1111/github/issues/Cockatrice/Cockatrice/App%20-%20Cockatrice.json ]:
Error: Error parsing JSON string: Unexpected token I in JSON at position 1
Given: [IcedFrisby] Destination URL may be down or URL is invalid, Error: ESOCKETTIMEDOUT

The "PHP version from .travis.yml" only seems to fail on the daily Circle CI test, unsure where it is failing though as 'invalid' is returned several places, guess it could be a rate limit against Circle CI's servers.

Answer 9 · 2017-12-28T19:24:15.000Z

That's a socket timeout presented by IcedFrisby in a strange way.

I'm going to re-run the daily tests and ssh into the build machine to see if I can diagnose further.

Answer 10 · 2017-12-28T19:36:33.000Z

Interesting; it's this one: https://github.com/badges/shields/blob/master/server.js#L336-L341

So the problem is that the request to the GitHub API is failing. Probably that is because it's not being sent with auth information, which also explains why the request usually works locally, but does not in CI. I'll open a PR with a fix.

Answer 11 · 2018-01-15T18:12:51.000Z

Took a quick pass through our failing service tests at https://circleci.com/gh/badges/shields/1355 and https://circleci.com/gh/badges/shields/1411.

This one seems to be failing consistently. Is it perhaps referring to an image that has gone away?

  10) MicroBadger
       image size without a specified tag
         
	[ GET http://localhost:1111/microbadger/image-size/_/hello-world.json ]:
     ValidationError: child "value" fails because ["value" with value "error" fails to match the required pattern: /^[0-9]*[.]?[0-9]+\s(B|kB|MB|GB|TB|PB|EB|ZB|YB)$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

The other failures look to be related to Github API (timeouts – not auth failures) and the Travis API.

I opened #1448 and #1449 to fix other failures.

Answer 12 · 2018-02-14T04:32:39.000Z

Current failing tests:

NPM package bundle size

  1) NPM package bundle size
       Testing format '/bundlephobia/:type/@:scope/:package.:format' against '/min/@cycle/core.json'
         
	[ GET http://localhost:1111/bundlephobia/min/@cycle/core.json ]:
     ValidationError: child "value" fails because ["value" with value "package not found error" fails to match the required pattern: /^[0-9]*[.]?[0-9]+\s(B|kB|MB|GB|TB|PB|EB|ZB|YB)$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  2) NPM package bundle size
       Testing format '/bundlephobia/:type/@:scope/:package.:format' against '/minzip/@cycle/core.json'
         
	[ GET http://localhost:1111/bundlephobia/minzip/@cycle/core.json ]:
     ValidationError: child "value" fails because ["value" with value "package not found error" fails to match the required pattern: /^[0-9]*[.]?[0-9]+\s(B|kB|MB|GB|TB|PB|EB|ZB|YB)$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

dotnet-status

  3) dotnet-status
       get nuget package status
         
	[ GET http://localhost:1111/dotnetstatus/gh/jaredcnance/dotnet-status/API.json ]:
     ValidationError: child "value" fails because ["value" must be one of [up to date, out of date, processing]]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  4) dotnet-status
       get nuget package status
         
	[ GET http://localhost:1111/dotnetstatus/gh/jaredcnance/dotnet-status/invalid-project.json ]:

      AssertionError: expected { Object (name, value) } to deeply equal { Object (name, value) }
      + expected - actual

       {
         "name": "dependencies"
      -  "value": "inconclusive"
      +  "value": "project not found"
       }
      
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38)
      at current.expects.push (node_modules/icedfrisby/lib/icedfrisby.js:724:10)
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

Github

  5) Github
       GitHub open issues by multi-word label is > zero
         
	[ GET http://localhost:1111/github/issues/Cockatrice/Cockatrice/App%20-%20Cockatrice.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  6) Github
       GitHub open issues by label (raw)
         
	[ GET http://localhost:1111/github/issues-raw/badges/shields/service-badge.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  7) Github
       GitHub open pull requests by label
         
	[ GET http://localhost:1111/github/issues-pr/badges/shields/service-badge.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  8) Github
       GitHub open pull requests by label (raw)
         
	[ GET http://localhost:1111/github/issues-pr-raw/badges/shields/service-badge.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  9) Github
       hit counter
         
	[ GET http://localhost:1111/github/search/torvalds/linux/goto.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  10) Github
       hit counter for nonexistent repo
         
	[ GET http://localhost:1111/github/search/torvalds/not-linux/goto.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

Gratipay - #1449

  11) Gratipay
       Receiving
         
	[ GET http://localhost:1111/gratipay/Gratipay.json ]:
     ValidationError: child "value" fails because ["value" with value "invalid" fails to match the required pattern: /^\$[0-9]+(\.[0-9]{2})?\/week/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

Hackage

  12) Hackage
       hackage version (valid)
         
	[ GET http://localhost:1111/hackage/v/lens.json ]:
" fails to match the required pattern: /^v\d+(\.\d+)?(\.\d+)?$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

Nexus

  13) Nexus
       search release version
         
	[ GET http://localhost:1111/nexus/r/https/repository.jboss.org/nexus/jboss/jboss-client.json ]:
     ValidationError: child "value" fails because ["value" with value "invalid" fails to match the required pattern: /^v4(\.\d+)+$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  14) Nexus
       search release version of an inexistent artifact
         
	[ GET http://localhost:1111/nexus/r/https/repository.jboss.org/nexus/jboss/inexistent-artifact-id.json ]:

      AssertionError: expected { name: 'nexus', value: 'invalid' } to deeply equal { name: 'nexus', value: 'no-artifact' }
      + expected - actual

       {
         "name": "nexus"
      -  "value": "invalid"
      +  "value": "no-artifact"
       }
      
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38)
      at current.expects.push (node_modules/icedfrisby/lib/icedfrisby.js:724:10)
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  15) Nexus
       search snapshot version
         
	[ GET http://localhost:1111/nexus/s/https/repository.jboss.org/nexus/com.progress.fuse/fusehq.json ]:
     ValidationError: child "value" fails because ["value" with value "invalid" fails to match the required pattern: /-SNAPSHOT$/]
      at Object.exports.process (node_modules/joi/lib/errors.js:190:19)
      at internals.Object._validateWithOptions (node_modules/joi/lib/types/any/index.js:669:31)
      at module.exports.internals.Any.root.validate (node_modules/joi/lib/index.js:139:23)
      at Object.pathMatch.matchJSONTypes (node_modules/icedfrisby/lib/pathMatch.js:303:9)
      at node_modules/icedfrisby/lib/icedfrisby.js:703:10
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  16) Nexus
       search snapshot version of a release artifact
         
	[ GET http://localhost:1111/nexus/s/https/repository.jboss.org/nexus/jboss/jboss-client.json ]:

      AssertionError: expected { name: 'nexus', value: 'invalid' } to deeply equal { name: 'nexus', value: 'undefined' }
      + expected - actual

       {
         "name": "nexus"
      -  "value": "invalid"
      +  "value": "undefined"
       }
      
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38)
      at current.expects.push (node_modules/icedfrisby/lib/icedfrisby.js:724:10)
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  17) Nexus
       search snapshot version of an inexistent artifact
         
	[ GET http://localhost:1111/nexus/s/https/repository.jboss.org/nexus/jboss/inexistent-artifact-id.json ]:

      AssertionError: expected { name: 'nexus', value: 'invalid' } to deeply equal { name: 'nexus', value: 'no-artifact' }
      + expected - actual

       {
         "name": "nexus"
      -  "value": "invalid"
      +  "value": "no-artifact"
       }
      
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38)
      at current.expects.push (node_modules/icedfrisby/lib/icedfrisby.js:724:10)
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1294:33)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1274:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1232:16)
      at Request.self.callback (node_modules/request/request.js:186:22)
      at Request.<anonymous> (node_modules/request/request.js:1163:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1085:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

PHP version from PHP-Eye

  18) PHP version from PHP-Eye
       gets the package version of symfony
         
	[ GET http://localhost:1111/php-eye/symfony/symfony.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  19) PHP version from PHP-Eye
       gets the package version of symfony 2.8
         
	[ GET http://localhost:1111/php-eye/symfony/symfony/v2.8.0.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  20) PHP version from PHP-Eye
       gets the package version of yii
         
	[ GET http://localhost:1111/php-eye/yiisoft/yii.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  21) PHP version from PHP-Eye
       invalid package name
         
	[ GET http://localhost:1111/php-eye/frodo/is-not-a-package.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

suggest

  22) suggest
       issues, forks, stars and twitter
         
	[ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  23) suggest
       license
         
	[ GET http://localhost:1111/$suggest/v1?url=https%3A%2F%2Fgithub.com%2Fatom%2Fatom ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

PHP version from .travis.yml

  24) PHP version from .travis.yml
       gets the package version of symfony
         
	[ GET http://localhost:1111/travis/php-v/symfony/symfony.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  25) PHP version from .travis.yml
       gets the package version of symfony 2.8
         
	[ GET http://localhost:1111/travis/php-v/symfony/symfony/2.8.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  26) PHP version from .travis.yml
       gets the package version of yii
         
	[ GET http://localhost:1111/travis/php-v/yiisoft/yii.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  27) PHP version from .travis.yml
       invalid package name
         
	[ GET http://localhost:1111/travis/php-v/frodo/is-not-a-package.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

Answer 13 · 2018-03-11T21:56:13.000Z

SonarQube has started failing again:

  16) SonarQube
       Tech Debt (legacy API unsupported)
         
	[ GET http://localhost:1111/sonar/4.2/http/sonarqube.com/com.github.dannil:scb-java-client/tech_debt.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

  17) SonarQube
       Coverage (legacy API unsupported)
         
	[ GET http://localhost:1111/sonar/4.2/http/sonarqube.com/com.github.dannil:scb-java-client/coverage.json ]:
     Error: Request timed out after 5000 ms
      at Timeout.handleRequestTimeout [as _onTimeout] (node_modules/icedfrisby/lib/icedfrisby.js:1199:35)

I suspect the repo is no longer valid for those tests.

Also the json badge is behaving strangely for this too:
SonarQube json badge

Answer 14 · 2018-03-11T23:12:22.000Z

Another possibility, since these particular tests are about what happens when the wrong API version is configured on the badge, is that the API being out of date is manifesting in a new way. I'm curious which of those it is.

Answer 15 · 2018-11-05T16:38:55.000Z

I expect this failure in NpmLicense is due to #2196:

https://circleci.com/gh/badges/shields/20848

Fixing an issue like this is a great way for a new contributor to jump in. Any thoughts on the best way to track these?

Answer 16 · 2018-11-17T20:58:30.000Z

This does not seem to be the best place to track these issues. I think it may be a good place to coordinate on the larger project, or figure out how to best support new contributors in addressing.

them. Let's default to opening new issues for these failures in the daily tests.