sharpart555/nexline

Error: Invalid input. Input must be one of these: string, buffer, readable stream, file descriptor

sbrl opened this issue · 2 comments

sbrl commented

Hello again!

Thanks for fixing that last bug. I suppose this one is more of a feature request really. Consider the following test program:

#!/usr/bin/env node
"use strict";

import fs from 'fs';

import gunzip from 'gunzip-maybe';
import nexline from 'nexline';

(async () => {
    "use strict";
    
	let handle_reader = fs.createReadStream("/tmp/test.txt");
	let extractor = gunzip();
	handle_reader.pipe(extractor);
	
	let line_reader = nexline({
		input: extractor // Could also be an array of inputs
	});
	
	console.log(`First line: ${await line_reader.next()}`);
})();

I'm trying to read a number of different (potentially) gzipped files line-by-line, but it looks like nexline gets upset by duplex streams (gunzip-maybe instances show Duplexify { .... } in when console.log()ged). It generates an error message like this:

(node:26326) UnhandledPromiseRejectionWarning: Error: Invalid input. Input must be one of these: string, buffer, readable stream, file descriptor
    at nexline (/home/bryan-smithl/Documents/repos/PhD-Code/ingester/node_modules/nexline/nexline.js:43:11)
    at multiInputWrapper (/home/bryan-smithl/Documents/repos/PhD-Code/ingester/node_modules/nexline/multiInputWrapper.js:25:7)
    at file:///home/bryan-smithl/Documents/repos/PhD-Code/ingester/backends/RadarCaesar/test.mjs:16:20
    at file:///home/bryan-smithl/Documents/repos/PhD-Code/ingester/backends/RadarCaesar/test.mjs:21:3
    at ModuleJob.run (internal/modules/esm/module_job.js:110:37)
    at async Loader.import (internal/modules/esm/loader.js:141:24)
(node:26326) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:26326) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

System info: Node.js version: v13.9.0, npm version: 6.13.7, Ubuntu 18.04.4 LTS

sharpart555 commented

Hi again 😃
nexline only supports nodejs standard Readable stream.
There are so many different types of custom streams but I think supporting them all seems undesirable.
I would suggest converting custom stream to nodejs standard stream like this.

#!/usr/bin/env node
"use strict";

import fs from 'fs';

import gunzip from 'gunzip-maybe';
import nexline from 'nexline';
import { Readable } from 'stream';

(async () => {
    "use strict";
    
	let handle_reader = fs.createReadStream("/tmp/test.txt");
	let extractor = gunzip();
	handle_reader.pipe(extractor);
        const rStream = new Readable().wrap(extractor);
	
	let line_reader = nexline({
		input: rStream,
	});
	
	console.log(`First line: ${await line_reader.next()}`);
})();
sbrl commented

Ah, I see! I wasn't aware that it wasn't a proper stream.

Thanks again!

  • 👍1