nomemory/mockneat

Change int to long for number of records passed to list...

rajib76 opened this issue · 4 comments

I will work on it and send a PR shortly. Currently the code takes only int for number of records. If we need billions of records created, we will need to change it to long

default MockUnit<List> list(Supplier<List> listSupplier, int size)

I am planning to add another method which will take long as an input.

It will work till 2,147,483,647. For more than this we will need long.

I don't think someone needs to keep in memory a list bigger than Integer max.

Hmm, I agree with you but should we look at any other option on how we can make it work if the need is to create large data let's say 3 billion records. May be some option to create the data in splits and write it in disk.

It's highly improbable someone would want to create a 3 billion records data set by keeping them in memory. Also a file that big is ... very big.

Still if you want to write data in chunks you can do it in a series of way:

For example:

 MockNeat m = MockNeat.threadLocal();

        int numStep = 10;
        int listSize = 1000;

        m.csvs()
         .column(m.intSeq())
         .column(m.names())
         .column(m.cities().capitals())
         .column(m.days())
         .column(m.creditCards().amex())
         .list(() -> new ArrayList<>(), listSize)
         .consume(numStep,  (step, list) -> {
             try { Files.write(Paths.get("test.csv"), list, CREATE, WRITE, APPEND); }
             catch (IOException e) { e.printStackTrace(); }
         });