bits-and-blooms/bitset

`bufio` in `WriteTo`/`ReadFrom` causes a lot of GC allocations

omerfirmak opened this issue · 0 comments

(pprof) omer@omer-ThinkPad-E14-Gen-3:~/Documents/juno$ go tool pprof -alloc_space http://localhost:9080/debug/pprof/heap
Fetching profile over HTTP from http://localhost:9080/debug/pprof/heap
Saved profile in /home/omer/pprof/pprof.juno.alloc_objects.alloc_space.inuse_objects.inuse_space.008.pb.gz
File: juno
Build ID: 951dffa00ff1dc027e7f19c22ffa115310c93eb0
Type: alloc_space
Time: Apr 22, 2023 at 6:09pm (+03)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 441.63GB, 93.58% of 471.93GB total
Dropped 677 nodes (cum <= 2.36GB)
Showing top 10 nodes out of 102
      flat  flat%   sum%        cum   cum%
  216.42GB 45.86% 45.86%   216.42GB 45.86%  bufio.NewWriterSize (inline)
  199.21GB 42.21% 88.07%   199.21GB 42.21%  bufio.NewReaderSize
    3.96GB  0.84% 88.91%     3.96GB  0.84%  reflect.New
    3.73GB  0.79% 89.70%     3.74GB  0.79%  io.ReadAll
    3.63GB  0.77% 90.47%     3.63GB  0.77%  github.com/cockroachdb/pebble.(*Batch).grow
    3.36GB  0.71% 91.18%     4.62GB  0.98%  bytes.(*Buffer).grow
    3.11GB  0.66% 91.84%     4.07GB  0.86%  encoding/json.(*decodeState).objectInterface
    3.09GB  0.65% 92.49%     3.09GB  0.65%  github.com/bits-and-blooms/bitset.New
    2.64GB  0.56% 93.05%     2.64GB  0.56%  github.com/cockroachdb/pebble.(*Batch).newInternalIter
    2.49GB  0.53% 93.58%   224.33GB 47.53%  github.com/bits-and-blooms/bitset.(*BitSet).MarshalBinary

Buffered counterparts of Reader and Writer interfaces allocate 4K each time.
bufio provides no benefit either, words are read individually in to a local buffer.