Runs outside of promise chain

Is there a way to await all retry attempts?

This simple test causes tests to fail (if you comment out nock part - to simulate a real world failure)

const nock = require('nock')
// Requiring our app implementation
const myProbotApp = require('..')
const { Probot } = require('probot')
// Requiring our fixtures
const payload = require('./fixtures/issues.opened')
const issueCreatedBody = { body: 'Thanks for opening this issue!' }

nock.disableNetConnect()

describe('My Probot app', () => {
  let probot

  beforeEach(() => {
    probot = new Probot({})
    // Load our app into probot
    const app = probot.load(myProbotApp)

    // just return a test token
    app.app = () => 'test'
  })

  test('creates a passing check', async () => {
    // Test that we correctly return a test token
    nock('https://api.github.com')
      .post('/app/installations/2/access_tokens')
      .reply(200, { token: 'test' })

    // Test that a commented is posted
    /* nock('https://api.github.com')
      .post('/repos/hiimbex/testing-things/issues/1/comments', (body) => {
        expect(body).toMatchObject(issueCreatedBody)
        return true
      })
      .reply(200) */

    // Receive a webhook event
    await probot.receive({ name: 'issues', payload })
  })
})

The error message looks something like this:

  ●  Cannot log after tests are done. Did you forget to wait for something async in your test?
    Attempted to log "{ HttpError: request to https://api.github.com/repos/hiimbex/testing-things/issues/1/comments failed, reason: Nock: Disallowed net connect for "api.github.com:443/repos/hiimbex/testing-things/issues/1/comments"
        at fetch.then.then.catch.error (/Users/pandah/git/pandah/lintbot/node_modules/@octokit/request/lib/request.js:94:13)
        at <anonymous>
        at process._tickDomainCallback (internal/process/next_tick.js:228:7)
      name: 'HttpError',
      status: 500,
      headers: {},
      request: 
       { method: 'POST',
         url: 'https://api.github.com/repos/hiimbex/testing-things/issues/1/comments',
         headers: 
          { accept: 'application/vnd.github.v3+json',
            'user-agent': 'octokit.js/16.25.0 Node.js/8.10.0 (macOS Mojave; x64)',
            authorization: 'token [REDACTED]' },
         request: { validate: [Object], retries: 3, retryAfter: 16, retryCount: 4 } } }".

      152 |                 expect(actualFailureStatus).toMatchObject(failureStatusBody)
      153 |             } catch (e) {
    > 154 |                 console.error(e)
          |                         ^
      155 |             }
      156 |         })
      157 |     })

      at fetch.then.then.catch.error (node_modules/@octokit/request/lib/request.js:94:13)
        name: 'HttpError',
        status: 500,
        headers: {},
        request: 
         { method: 'POST',
           url: 'https://api.github.com/repos/hiimbex/testing-things/issues/1/comments',
           headers: 
            { accept: 'application/vnd.github.v3+json',
              'user-agent': 'octokit.js/16.25.0 Node.js/8.10.0 (macOS Mojave; x64)',
              authorization: 'token [REDACTED]' },
           request: { validate: [Object], retries: 3, retryAfter: 16, retryCount: 4 } } }".
      at Object.error (test/index.test.js:154:25)

Am I missing something?

A lot is going on here that is not plugin-retry.js. If possible, try to reduce the test case to only use Octokit with the retry plugin.

In your case, I think you can pass a custom Octokit instance without the retry logic:

probot = new Probot({
  Octokit: require('@octokit/rest') // use bare @octokit/rest, without any plugins
})

Using

probot = new Probot({
  Octokit: require('@octokit/rest') // use bare @octokit/rest, without any plugins
})

the tests fail (as expected) but the promise violation no longer happens.

So, can we close this issue? If there is still an issue, I’d suggest to open an issue in Probot or create a test case which does not include Probot. Because I’m fairly sure that plugin-retry.js does not settle request promises while the request is still being retried.

@gr2m I guess plugin-retry.js is returning the promise

plugin-retry.js/lib/wrap-request.js

Line 20 in de1092a

return limiter.schedule(request, options)

but I'm not sure where the issue lies in the pipeline.

Someone somewhere in the chain is "dropping the ball" (so-to-speak) - either by not awaiting or not returning the promise. I can create an issue in Probot.

I can create an issue in Probot

Thanks!

One more thought I had: can you try to run tests with the environmental variable DISABLE_STATS set to true to disable stats? See https://probot.github.io/docs/configuration/

Let me try that and get back to you.

let me know if the problem persists