caolan/highland

How to terminate a generator should be documented

abeluck opened this issue · 0 comments

The docs say this about generators:

Generators - These are functions which provide values for the Stream. They are lazy and can be infinite, they can also be asynchronous (for example, making a HTTP request). You emit values on the Stream by calling push(err, val), much like a standard Node.js callback. Once it has been called, the generator function will not be called again unless you call next(). This call to next() will signal you've finished processing the current data and allow for the generator function to be called again. If the Stream is still being consumed the generator function will then be called again.
You can also redirect a generator Stream by passing a new source Stream to read from to next. For example: next(other_stream) - then any subsequent calls will be made to the new source.

I implemented the described interface. My generator is paging through HTTP responses. Once the pages were exhausted however the final consumption functions (like collect() or toArray() or toPromise()) were not being called.

I struggled for some time, thinking the problem was with my stream pipeline somewhere.

Only later did I discover that the stream source itself was still open waiting for a call to next(). By looking at other issues in this repo I eventually discovered the highland.nil "end of stream marker" Adding push(null, highland.nil) to my generator once the pages were exhausted instantly closed the stream and my problem was resolved.

This may seem obvious to those familiar with highland, but it was totally a surprise to me. Also I realize highland.nil clearly states it is the end of stream marker, but I hadn't browsed that part of the docs. Ideally I think the generator docs should mention something about terminating itself.

Finally.. before I added highland.nil my generator actually had no dependency on highland itself, which was quite nice. Now however it does. Is there a way to terminate the generator without requiring highland in my generator module?