/large-csv-stream-concat-nodejs

Node.js script that concatenates unlimited CSV's into a single file using Streams.

Primary LanguageJavaScript

Large CSV Concat with Node.js Streams

👷 Developed by Matheus Ramalho de Oliveira
🔨 Brazilian Software Engineer
🏡 Goiânia, Goiás, Brasil
✉️ kastorcode@gmail.com
👍 instagram.com/kastorcode

This script uses the best possible practice for working with files on demand in Node.js. We will concatenate unlimited gigabytes of information from different files converting from CSV to JSON, taking the data we are interested in and converting it back from JSON to CSV, all using streams.

Installation and execution

  1. Make a clone of this repository;
  2. Create a folder called dataset;
  3. Download 2 data packs: https://kaggle.com/datasets/stackoverflow/so-survey-2017 and https://kaggle.com/datasets/stackoverflow/stack-overflow-2018-developer-survey;
  4. Extract the data into the dataset folder;
  5. Open the project folder in a terminal;
  6. Run yarn to install dependencies;
  7. Run yarn start to start the concatenation.

<kastor.code/>