Reading a very large file

[saha-kpghoshal]

Hi @yortus

I was having trouble reading for a large file.
dbf.readRecords(dbf.recordCount) results in a out of memory exception,
could you suggest a way to iterate through the whole file in small chunks

[yortus]

The API is designed to read DBF files in any chunk size you choose. In your code above you are asking to read the entire set of records in one go, hence the memory error.

Try reading 100 or 1000 rows at a time, process those rows, then read the next chunk of rows, and so on until you have read all dbf.recordCount rows. You can do it in a loop inside an async function, something like this:

async function processWholeFile() {
    let dbf = await DBFFile.open('path/to/file.dbf');
    let batchSize = 500;
    let rowsRemaining = dbf.recordCount;

    while (rowsRemaining > 0) {
        let rowsToRead = rowsRemaining > batchSize ? batchSize : rowsRemaining;
        let rows = await dbf.readRecords(rowsToRead);
        rowsRemaining -= rowsToRead;

        rows.forEach(row => {
            // ...process these rows...
        });
    }
}

[saha-kpghoshal]

Thanks a lot, for your help.