When piping to the mock response, finish/end callback is never called

[alexgurr]

Handler code:

function getDownloadData(downloadInfo, res) {
    request.post({ url: downloadInfo.url, formData: { download_reference: downloadInfo.token }}, (err, rideResponse) => {
        if(rideResponse.headers['content-type'].includes('text/xml')) {
            logError(`Failed to get audio data from RIDE service: ${rideResponse.body}`);
        }
    })
    .pipe(res);
}

I see console logs for 'DATA, and 'RESPONSE', but the callback on my mock response is never called. I know this works, because I can pipe to a real response object, and get the results I expect in my client.

Code that handles the POST:

app.post('/ride/anndownload', (req, res) => {
    log.info('RIDE Announcement Stub :: Got request to download announcement - returning download');
    const file = './src/stubs/resources/example_ann_wav.wav';

    res.download(file);
});

My test use of mockRes is simply:

   const finish() {
      console.log('finished!');
   }
   const mockRes = new MockRes(finish);

Then I call my handler directly with this response object. The finished console log is never called, and the test times out.

Any ideas?

[diachedelic]

Ah that's interesting!

Looking at the docs, http.ServerResponse only emits finish, not end.

Does that help?

Edit: wait - if the stream you pipe into your res does not call end, I don't think the res should emit finish - can you confirm that console.log('END!') is being called?

[alexgurr]

The series of '.on' functions are only there for debugging. I can quite easily remove them - same with the mockRes.on('end') - it's only there for trying random things to get it to work. None of the mockRes on functions are called.

[diachedelic]

Ah quick reply, my edit:

if the stream you pipe into your res does not call end, I don't think the res should emit finish - can you confirm that console.log('END!') is being called?

[alexgurr]

End doesn't seem to be being called. I see RESPONSE! DATA! DATA!, then nothing.

When I switch on my error scenario (where the POST response gets responded to with an error response, and is not a download response), mockRes seems to work. I see Response! Data! End! Thi just seems to be an issue when I respond with a download.

Interestingly, when I test this in a real life scenario, with a real response to the browser, it works fine.

[diachedelic]

Hmm, even downloads should end.

[diachedelic]

I would investigate why res.download(file); is not calling end if I were you!

[alexgurr]

Surely if this works with a real response object (and end is called), but not mockRes - this is a potential mockRes issue?

[alexgurr]

Same requests (first one from browser, second using mockRes):

https://puu.sh/uB5t8/57cc47ced3.png

https://puu.sh/uB5ui/324e03e6bc.png

Second times out. What's interesting is comparing the data chunks:

https://puu.sh/uB5OL/85df7f8562.png

https://puu.sh/uB5Pr/e751cef884.png

The first data chunk matches, but then it's like something happens to the stream.

[diachedelic]

Now, I may be wrong, but i think it works like this -

You pipe a stream implementing 'Readable' into a stream implementing 'Writable'. Readable stream (request.post()) indicates to Writable stream (res) it is done sending data by emitting event end. Upon finishing up, the Writable stream emits finish.

So perhaps one of these is happening:

1. Writable is somehow blocking Readable, so it can't push all its data and emit end.
2. Readable is not calling end for some other reason.
3. Readable is calling end but Writable is not calling finish.

I've made sure (3) is not the case by writing a test case: (see commit ef3797b)

I can't see (1) being the issue...MockRes doesn't apply any restrictions to how much data it can accept.

So I'm thinking it's (2) - it's up to your Readable to call end.

[diachedelic]

Stricter test: https://github.com/diachedelic/mock-res/blob/master/test.js#L173

[diachedelic]

I have come across plenty of situations where end is not called for very weird reasons.

[alexgurr]

It's not calling end because the data hasn't finished being piped. If you look at the 4 images above, you can see the number of data chunks doesn't match. The second chunk is incorrect/corrupt, and then nothing happens (when using mockRes)...

It's almost like there's an issue with the stream. If this was consistent in the test and in the browser, I could understand there's an issue. But what's weird is for me, is why this only has this behaviour when testing.

I put a console.log in _transform inside index.js. Only the first chunk gets passed into this function. The second logged data chunk (by express), never gets passed or logged in _transform, hence why we never see any subsequent chunks logged by express (because it seems to fail to pass the second chunk of data)

[diachedelic]

I really think you'll have to find out why your readable stream isn't emitting all its data.

If you are doing a unit test (mock-res isn't for integration tests) then how can you be sure res.download() is doing it's job correctly without the whole app running?

[alexgurr]

There's 3 layers. Client <=> Server <=> Stub. There's tests to make sure the download from the stub works. Im now testing the handler at the server level, to make sure the handler handles the download properly (by mocking the res object, and directly injecting req/res to the handler). I could mock the download, but that's basically what the stub does anyway.

[diachedelic]

Well try this - I don't htink you're meant to use a callback AND stream your request.post() call. From docs:

request
  .get('http://google.com/img.png')
  .on('response', function(response) {
    console.log(response.statusCode) // 200 
    console.log(response.headers['content-type']) // 'image/png' 
  })
  .pipe(request.put('http://mysite.com/img.png'))

[diachedelic]

To get error:

request
  .get('http://mysite.com/doodle.png')
  .on('error', function(err) {
    console.log(err)
  })
  .pipe(fs.createWriteStream('doodle.png'))