Change split from bytes to lines
Not-C-Developer opened this issue · 2 comments
Not-C-Developer commented
Hi.
Change this.
from
split -b100M rdns.rev.lowercase.txt fileChunk
to
split -l2000000 rdns.rev.lowercase.txt fileChunk
in scripts/fdns_a.sh and scripts/rdns.sh
because lose some records when sorting.
j0eii commented
Yes true, it has data loss.
it fixed my issue too.
redNixon commented
From split's man page:
-C, --line-bytes=SIZE
put at most SIZE bytes of records per output file
Seems like that would be the best option to use to give you a desired file chunk size without risking data loss.