This Node.js library is designed to export millions of records from a MongoDB database. The library follows a multi-step process, starting with creating and inserting documents into the MongoDB database and then capturing the documents as a MongoDB stream, ultimately exporting them to a CSV file.
To use this library, follow these steps:
-
Clone the repository to your local machine.
-
Install the required dependencies by running the following command:
pnpm install
-
Run following commands
npm run compile npm run seed // to create dummy csv file for demo purpose (optional) npm start // to insert records in db in batches
Set up your environment by creating a .env file with the necessary MongoDB connection URI. Make sure to define the DB_URI environment variable in this file.
Before using the library, make sure to configure it properly:
- `FILE_NAME`: The name of the CSV file to which the data will be exported.
- `BATCH_SIZE`: The number of records to be inserted into MongoDB at once.
The library uses the MongoDB Node.js driver to connect to the MongoDB database.
The CSV data is read from the FILE_NAME file and transformed using a CSVTOJSON transformation stream. A very important transformation to read and tranform stream.
The transformed data is batched and inserted into the MongoDB collection 'users' within the 'mydb1' database. You can change the db and collection name according to the needs.
The library provides detailed logging of the data transformation and insertion process.
seed.ts file use to export the file into csv in order to insert into mongodb.
This library is provided under the MIT License. Feel free to modify and use it according to your requirements.
This library is created and maintained by Neeraj Kumar. For any questions or issues, please contact on email ennkay161@gmail.com.
Special thanks to the MongoDB Node.js driver and other libraries used in this project.