Large CSV files with newlines are incorrectly splitted for upsert operations

Question

Large CSV files with newlines are incorrectly splitted for upsert operations

Closed this issue 7 years ago · 3 comments

force version: v0.22.66

The split of large csv files for data loading operations is generating the batches splitting the source CSV file by rows instead of records, because if a file has newline characters in it (i.e. Long or rich text fields), the record uses more than one line in the file.

Batches then are incorrectly generated as they obviously don't meet the expected structure.

You can recreate this simply by creating a large csv file with newlines and insert/upsert/update-ing it on an instance.

Answer 1 · 2017-09-22T15:44:26.000Z

We should use encoding/csv to read in the csv. splitFileIntoBatches is simply splitting on newlines.

Answer 2 · 2017-10-09T17:04:08.000Z

I've opened #452 to fix parsing of multi-line values in csv.

Answer 3 · 2017-11-29T22:53:48.000Z

Fixed in v0.22.81