sharpart555/nexline

Read file, stream, string, buffer line by line without putting them all in memory. It supports cool features like `custom line separator`, `various encodings`, `reverse mode`, `iterable protocol`

Nexline

Read file, stream, string, buffer line by line without putting them all in memory.
It supports cool features below.

• Lightweight
• Iterable protocol
• Handle large file with small memory footprint
• Support various input types (file, stream, string, buffer)
• Support various encodings
• Support reverse mode
• Support custom, multiple line separators
• Support multiple inputs

Why I made this?

Node.js's default readline module is great but it's pause() method does not work immediately.
I made wrapper to fix that issue before, but it was not enough.
Even if pause() works immediately, it is still inconvenient.
If I want to execute async function over line by line in large file, I have to call pause() and resume() at every line.
I needed better way to do that without putting them all in memory.

Breaking changes in 1.0.0

• lineSeparator default value is changed from ['\n', '\r\n'] to '\n' for better performance
  • If you want to handle both CRLF and LF, set lineSeparator: ['\n', '\r\n']
• Support reverse mode.
• Support file descriptor as input
• Improve performance
• Optimize memory usage

Install with npm

Required Node.js version >= 8.0.0

npm install nexline

How to use

Use next() method

const nexline = require('nexline');

async function main () {
  const nl = nexline(...);
  
  while(true) {
    const line = await nl.next();
    console.log(line);
    if (line === null) break; // If all data is read, returns null
  }
}

Use as iterator ( Requires node >= 10 )

const nexline = require('nexline');

async function main () {
  const nl = nexline(...);

  // nexline is iterable
  for await (const line of nl) console.log(line);
}

Use file as input

• Don't forget to close file descriptor after finish
• You can use autoCloseFile: true to close file descriptor automatically

const nl = nexline({
  input: fs.openSync(path_to_file, 'r'),
  // autoCloseFile: true,
});

Use stream as input

const nl = nexline({
  input: fs.createReadStream(path_to_file),
});

Use string as input

const nl = nexline({
  input: 'foo\nbar\nbaz',
});

Use buffer as input

const nl = nexline({
  input: Buffer.from('foo\nbar\nbaz'),
});

Use multiple input

const nl = nexline({
  input: [
    fs.openSync(path_to_file1, 'r'),
    fs.createReadStream(path_to_file2),
    'foo\nbar\nbaz',
    Buffer.from('foo\nbar\nbaz'),
  ],
});

Use other encodings

See encodings supported by iconv-lite

const nl = nexline({
  input: fs.openSync(path_to_file, 'r'), 
  encoding: 'cp949',
});

Use other lineSeparator

const nl = nexline({
  input: 'foo;bar;baz', 
  lineSeparator: ';',
});

Use multiple lineSeparator

const nl = nexline({
  input: 'foo\r\nbar\nbaz', 
  lineSeparator: ['\n', '\r\n'], // You can handle both LF and CRLF like this.
});

Reverse mode

const nl = nexline({
  input: fs.openSync(path_to_file, 'r'), // NOTE: You cannot use stream in reverse mode. 
  reverse: true, 
});

Methods

Name	Description
next()	async, It returns line until all data is read, after then returns null
close()	Close nexline

Options

Name	Default	Description
input	undefined	Required. File descriptor, stream, string, buffer, You can provide multiple inputs using array
lineSeparator	'\n'	Any string more than one character. You can provide multiple line separator using array
encoding	'utf8'	See encodings supported by iconv-lite
reverse	false	Reverse mode, Cannot use this option with stream input because stream cannot be read from end. Use file descriptor instead
autoCloseFile	false	Automatically close file descriptor after all data is read

Contribution

Pull requests and stars are always welcome. For bugs and feature requests, please create an issue

License

Copyright © 2019 Youngho Jin. Released under the MIT License