Memory rise up when use writer.Write(data) and will not be GC
Closed this issue · 1 comments
yorsita commented
Hi team, I found the memory usage will rise up when I called writer.Write(data) and will not be GC even I did called writer.Close()
I tried to enable pprof and found there are lots of writer buffer object in the alloc: (below are some of my alloc record)
0: 0 [2: 4227072] @ 0x102dc47 0x102dabe 0x1031ab1 0x103319f 0x103276c 0x10a2fa3 0x10a6898 0x10a0ec5 0x46aa01
# 0x102dc46 github.com/segmentio/parquet-go.(*byteArrayColumnBuffer).append+0x1e6 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer.go:1510
# 0x102dabd github.com/segmentio/parquet-go.(*byteArrayColumnBuffer).writeValues+0x5d /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer.go:1483
# 0x1031ab0 github.com/segmentio/parquet-go.writeRowsFuncOfRequired.func1+0x50 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer_go18.go:77
# 0x103319e github.com/segmentio/parquet-go.writeRowsFuncOfStruct.func2+0xfe /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer_go18.go:326
# 0x103276b github.com/segmentio/parquet-go.writeRowsFuncOfPointer.func1+0x12b /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer_go18.go:202
# 0x10a2fa2 github.com/segmentio/parquet-go.makeWriteFunc[...].func1+0x122 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:130
# 0x10a6897 github.com/segmentio/parquet-go.(*GenericWriter[...]).Write+0x37 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:151
# 0x10a0ec4 feed-up-market/internal/feedup.(*FeedUpStrategy).saveBBOs+0x324 /home/test/carol/feed-up-market/internal/feedup/strategy.go:327
0: 0 [1: 950272] @ 0x102dbb7 0x102dabe 0x1031ab1 0x103319f 0x103276c 0x10a2fa3 0x10a6898 0x10a0ec5 0x46aa01
# 0x102dbb6 github.com/segmentio/parquet-go.(*byteArrayColumnBuffer).append+0x156 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer.go:1509
# 0x102dabd github.com/segmentio/parquet-go.(*byteArrayColumnBuffer).writeValues+0x5d /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer.go:1483
# 0x1031ab0 github.com/segmentio/parquet-go.writeRowsFuncOfRequired.func1+0x50 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer_go18.go:77
# 0x103319e github.com/segmentio/parquet-go.writeRowsFuncOfStruct.func2+0xfe /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer_go18.go:326
# 0x103276b github.com/segmentio/parquet-go.writeRowsFuncOfPointer.func1+0x12b /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer_go18.go:202
# 0x10a2fa2 github.com/segmentio/parquet-go.makeWriteFunc[...].func1+0x122 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:130
# 0x10a6897 github.com/segmentio/parquet-go.(*GenericWriter[...]).Write+0x37 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:151
# 0x10a0ec4 feed-up-market/internal/feedup.(*FeedUpStrategy).saveBBOs+0x324 /home/test/carol/feed-up-market/internal/feedup/strategy.go:327
0: 0 [1: 950272] @ 0x102db25 0x102dabe 0x1031ab1 0x103319f 0x103276c 0x10a2fa3 0x10a6898 0x10a0ec5 0x46aa01
# 0x102db24 github.com/segmentio/parquet-go.(*byteArrayColumnBuffer).append+0xc4 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer.go:1508
# 0x102dabd github.com/segmentio/parquet-go.(*byteArrayColumnBuffer).writeValues+0x5d /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer.go:1483
# 0x1031ab0 github.com/segmentio/parquet-go.writeRowsFuncOfRequired.func1+0x50 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer_go18.go:77
# 0x103319e github.com/segmentio/parquet-go.writeRowsFuncOfStruct.func2+0xfe /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer_go18.go:326
# 0x103276b github.com/segmentio/parquet-go.writeRowsFuncOfPointer.func1+0x12b /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/column_buffer_go18.go:202
# 0x10a2fa2 github.com/segmentio/parquet-go.makeWriteFunc[...].func1+0x122 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:130
# 0x10a6897 github.com/segmentio/parquet-go.(*GenericWriter[...]).Write+0x37 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:151
# 0x10a0ec4 feed-up-market/internal/feedup.(*FeedUpStrategy).saveBBOs+0x324 /home/test/carol/feed-up-market/internal/feedup/strategy.go:327
0: 0 [1: 576] @ 0x4f3285 0x4f2bdf 0x4f3492 0x101d2fc 0x101ea45 0x1020cbe 0x102178f 0x1020d10 0x1020078 0x1020d10 0x102178f 0x101f22a 0x101eea6 0x107cbc5 0x107c74f 0x10a0f51 0x10a0f3f 0x10a0f3e 0x46aa01
# 0x4f3284 bytes.makeSlice+0x64 /usr/local/go/src/bytes/buffer.go:229
# 0x4f2bde bytes.(*Buffer).grow+0x11e /usr/local/go/src/bytes/buffer.go:142
# 0x4f3491 bytes.(*Buffer).WriteByte+0x51 /usr/local/go/src/bytes/buffer.go:267
# 0x101d2fb github.com/segmentio/encoding/thrift.(*binaryWriter).writeByte+0x7b /home/test/go/pkg/mod/github.com/segmentio/encoding@v0.3.5/thrift/binary.go:358
# 0x101ea44 github.com/segmentio/encoding/thrift.(*compactWriter).WriteField+0x84 /home/test/go/pkg/mod/github.com/segmentio/encoding@v0.3.5/thrift/compact.go:309
# 0x1020cbd github.com/segmentio/encoding/thrift.(*structEncoder).encode+0x3fd /home/test/go/pkg/mod/github.com/segmentio/encoding@v0.3.5/thrift/encode.go:302
# 0x102178e github.com/segmentio/encoding/thrift.encodeFuncPtrOf.func1+0xee /home/test/go/pkg/mod/github.com/segmentio/encoding@v0.3.5/thrift/encode.go:397
# 0x1020d0f github.com/segmentio/encoding/thrift.(*structEncoder).encode+0x44f /home/test/go/pkg/mod/github.com/segmentio/encoding@v0.3.5/thrift/encode.go:307
# 0x1020077 github.com/segmentio/encoding/thrift.encodeFuncSliceOf.func1+0x157 /home/test/go/pkg/mod/github.com/segmentio/encoding@v0.3.5/thrift/encode.go:158
# 0x1020d0f github.com/segmentio/encoding/thrift.(*structEncoder).encode+0x44f /home/test/go/pkg/mod/github.com/segmentio/encoding@v0.3.5/thrift/encode.go:307
# 0x102178e github.com/segmentio/encoding/thrift.encodeFuncPtrOf.func1+0xee /home/test/go/pkg/mod/github.com/segmentio/encoding@v0.3.5/thrift/encode.go:397
# 0x101f229 github.com/segmentio/encoding/thrift.(*Encoder).Encode+0x2e9 /home/test/go/pkg/mod/github.com/segmentio/encoding@v0.3.5/thrift/encode.go:49
# 0x101eea5 github.com/segmentio/encoding/thrift.Marshal+0xa5 /home/test/go/pkg/mod/github.com/segmentio/encoding@v0.3.5/thrift/encode.go:19
# 0x107cbc4 github.com/segmentio/parquet-go.(*writer).writeFileFooter+0x284 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:517
# 0x107c74e github.com/segmentio/parquet-go.(*writer).close+0x4e /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:439
# 0x10a0f50 github.com/segmentio/parquet-go.(*Writer).Close+0x3b0 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:98
# 0x10a0f3e github.com/segmentio/parquet-go.(*GenericWriter[...]).Close+0x39e /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:139
# 0x10a0f3d feed-up-market/internal/feedup.(*FeedUpStrategy).saveBBOs+0x39d /home/test/carol/feed-up-market/internal/feedup/strategy.go:329
0: 0 [2: 8077312] @ 0x4f3285 0x4f2bdf 0x4f2e46 0x1080c0e 0x1081337 0x10809cb 0x107f60f 0x10a6925 0x10a0ec5 0x46aa01
# 0x4f3284 bytes.makeSlice+0x64 /usr/local/go/src/bytes/buffer.go:229
# 0x4f2bde bytes.(*Buffer).grow+0x11e /usr/local/go/src/bytes/buffer.go:142
# 0x4f2e45 bytes.(*Buffer).Write+0x65 /usr/local/go/src/bytes/buffer.go:172
# 0x1080c0d github.com/segmentio/parquet-go.(*writerColumn).writeDataPage.func1+0x14d /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1149
# 0x1081336 github.com/segmentio/parquet-go.(*writerColumn).writePageTo+0xf6 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1221
# 0x10809ca github.com/segmentio/parquet-go.(*writerColumn).writeDataPage+0x78a /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1142
# 0x107f60e github.com/segmentio/parquet-go.(*writerColumn).flush+0xae /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:890
# 0x10a6924 github.com/segmentio/parquet-go.(*GenericWriter[...]).Write+0xc4 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:160
# 0x10a0ec4 feed-up-market/internal/feedup.(*FeedUpStrategy).saveBBOs+0x324 /home/test/carol/feed-up-market/internal/feedup/strategy.go:327
0: 0 [1: 4235264] @ 0x4f3285 0x4f2bdf 0x4f2e46 0x1080c0e 0x1081337 0x10809cb 0x107f60f 0x10a6925 0x10a0ec5 0x46aa01
# 0x4f3284 bytes.makeSlice+0x64 /usr/local/go/src/bytes/buffer.go:229
# 0x4f2bde bytes.(*Buffer).grow+0x11e /usr/local/go/src/bytes/buffer.go:142
# 0x4f2e45 bytes.(*Buffer).Write+0x65 /usr/local/go/src/bytes/buffer.go:172
# 0x1080c0d github.com/segmentio/parquet-go.(*writerColumn).writeDataPage.func1+0x14d /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1149
# 0x1081336 github.com/segmentio/parquet-go.(*writerColumn).writePageTo+0xf6 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1221
# 0x10809ca github.com/segmentio/parquet-go.(*writerColumn).writeDataPage+0x78a /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1142
# 0x107f60e github.com/segmentio/parquet-go.(*writerColumn).flush+0xae /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:890
# 0x10a6924 github.com/segmentio/parquet-go.(*GenericWriter[...]).Write+0xc4 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:160
# 0x10a0ec4 feed-up-market/internal/feedup.(*FeedUpStrategy).saveBBOs+0x324 /home/test/carol/feed-up-market/internal/feedup/strategy.go:327
0: 0 [1: 4325376] @ 0x4f3285 0x4f2bdf 0x4f2e46 0x1080c0e 0x1081337 0x10809cb 0x107f60f 0x10a6925 0x10a0ec5 0x46aa01
# 0x4f3284 bytes.makeSlice+0x64 /usr/local/go/src/bytes/buffer.go:229
# 0x4f2bde bytes.(*Buffer).grow+0x11e /usr/local/go/src/bytes/buffer.go:142
# 0x4f2e45 bytes.(*Buffer).Write+0x65 /usr/local/go/src/bytes/buffer.go:172
# 0x1080c0d github.com/segmentio/parquet-go.(*writerColumn).writeDataPage.func1+0x14d /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1149
# 0x1081336 github.com/segmentio/parquet-go.(*writerColumn).writePageTo+0xf6 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1221
# 0x10809ca github.com/segmentio/parquet-go.(*writerColumn).writeDataPage+0x78a /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1142
# 0x107f60e github.com/segmentio/parquet-go.(*writerColumn).flush+0xae /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:890
# 0x10a6924 github.com/segmentio/parquet-go.(*GenericWriter[...]).Write+0xc4 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:160
# 0x10a0ec4 feed-up-market/internal/feedup.(*FeedUpStrategy).saveBBOs+0x324 /home/test/carol/feed-up-market/internal/feedup/strategy.go:327
0: 0 [1: 4186112] @ 0x4f3285 0x4f2bdf 0x4f2e46 0x1080c0e 0x1081337 0x10809cb 0x107f60f 0x10a6925 0x10a0ec5 0x46aa01
# 0x4f3284 bytes.makeSlice+0x64 /usr/local/go/src/bytes/buffer.go:229
# 0x4f2bde bytes.(*Buffer).grow+0x11e /usr/local/go/src/bytes/buffer.go:142
# 0x4f2e45 bytes.(*Buffer).Write+0x65 /usr/local/go/src/bytes/buffer.go:172
# 0x1080c0d github.com/segmentio/parquet-go.(*writerColumn).writeDataPage.func1+0x14d /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1149
# 0x1081336 github.com/segmentio/parquet-go.(*writerColumn).writePageTo+0xf6 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1221
# 0x10809ca github.com/segmentio/parquet-go.(*writerColumn).writeDataPage+0x78a /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1142
# 0x107f60e github.com/segmentio/parquet-go.(*writerColumn).flush+0xae /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:890
# 0x10a6924 github.com/segmentio/parquet-go.(*GenericWriter[...]).Write+0xc4 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:160
# 0x10a0ec4 feed-up-market/internal/feedup.(*FeedUpStrategy).saveBBOs+0x324 /home/test/carol/feed-up-market/internal/feedup/strategy.go:327
0: 0 [1: 1613824] @ 0x1008f5f 0xf838ef 0x10715cd 0x107f0b4 0x1080365 0x107f60f 0x10a6925 0x10a0ec5 0x46aa01
# 0x1008f5e github.com/segmentio/parquet-go/encoding/delta.(*LengthByteArrayEncoding).EncodeByteArray+0x1de /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/encoding/delta/length_byte_array.go:33
# 0xf838ee github.com/segmentio/parquet-go/encoding.EncodeByteArray+0x8e /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/encoding/values.go:265
# 0x10715cc github.com/segmentio/parquet-go.(*stringType).Encode+0x6c /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/type.go:931
# 0x107f0b3 github.com/segmentio/parquet-go.(*writerBuffers).encode+0x113 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:773
# 0x1080364 github.com/segmentio/parquet-go.(*writerColumn).writeDataPage+0x124 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1061
# 0x107f60e github.com/segmentio/parquet-go.(*writerColumn).flush+0xae /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:890
# 0x10a6924 github.com/segmentio/parquet-go.(*GenericWriter[...]).Write+0xc4 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:160
# 0x10a0ec4 feed-up-market/internal/feedup.(*FeedUpStrategy).saveBBOs+0x324 /home/test/carol/feed-up-market/internal/feedup/strategy.go:327
0: 0 [1: 1286144] @ 0x1008f5f 0xf838ef 0x10715cd 0x107f0b4 0x1080365 0x107f60f 0x10a6925 0x10a0ec5 0x46aa01
# 0x1008f5e github.com/segmentio/parquet-go/encoding/delta.(*LengthByteArrayEncoding).EncodeByteArray+0x1de /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/encoding/delta/length_byte_array.go:33
# 0xf838ee github.com/segmentio/parquet-go/encoding.EncodeByteArray+0x8e /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/encoding/values.go:265
# 0x10715cc github.com/segmentio/parquet-go.(*stringType).Encode+0x6c /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/type.go:931
# 0x107f0b3 github.com/segmentio/parquet-go.(*writerBuffers).encode+0x113 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:773
# 0x1080364 github.com/segmentio/parquet-go.(*writerColumn).writeDataPage+0x124 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1061
# 0x107f60e github.com/segmentio/parquet-go.(*writerColumn).flush+0xae /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:890
# 0x10a6924 github.com/segmentio/parquet-go.(*GenericWriter[...]).Write+0xc4 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:160
# 0x10a0ec4 feed-up-market/internal/feedup.(*FeedUpStrategy).saveBBOs+0x324 /home/test/carol/feed-up-market/internal/feedup/strategy.go:327
0: 0 [1: 1458176] @ 0x1008e7e 0x1008e4a 0xf838ef 0x10715cd 0x107f0b4 0x1080365 0x107f60f 0x10a6925 0x10a0ec5 0x46aa01
# 0x1008e7d github.com/segmentio/parquet-go/encoding/delta.(*int32Buffer).resize+0xfd /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/encoding/delta/delta.go:16
# 0x1008e49 github.com/segmentio/parquet-go/encoding/delta.(*LengthByteArrayEncoding).EncodeByteArray+0xc9 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/encoding/delta/length_byte_array.go:28
# 0xf838ee github.com/segmentio/parquet-go/encoding.EncodeByteArray+0x8e /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/encoding/values.go:265
# 0x10715cc github.com/segmentio/parquet-go.(*stringType).Encode+0x6c /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/type.go:931
# 0x107f0b3 github.com/segmentio/parquet-go.(*writerBuffers).encode+0x113 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:773
# 0x1080364 github.com/segmentio/parquet-go.(*writerColumn).writeDataPage+0x124 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:1061
# 0x107f60e github.com/segmentio/parquet-go.(*writerColumn).flush+0xae /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer.go:890
# 0x10a6924 github.com/segmentio/parquet-go.(*GenericWriter[...]).Write+0xc4 /home/test/go/pkg/mod/github.com/segmentio/parquet-go@v0.0.0-20221013173504-719054139246/writer_go18.go:160
# 0x10a0ec4 feed-up-market/internal/feedup.(*FeedUpStrategy).saveBBOs+0x324 /home/test/carol/feed-up-market/internal/feedup/strategy.go:327
here is the sample code:
package main
import (
"bytes"
"fmt"
"io"
"log"
"net/http"
_ "net/http/pprof"
"os"
"runtime"
"time"
cmap "github.com/orcaman/concurrent-map/v2"
"github.com/segmentio/parquet-go"
)
func main() {
go func() {
endpoint := "0.0.0.0:8000"
fmt.Printf("enable pprof at port %s\n", endpoint)
err := http.ListenAndServe(endpoint, nil)
if err != nil {
fmt.Printf("error occurred %v\n", err)
}
}()
ticker := time.NewTicker(100 * time.Millisecond)
for {
select {
case t := <-ticker.C:
TestStore(t.Format(time.StampMilli))
}
}
}
func TestStore(name string) {
type People struct {
Name string
Age int
}
type Nested struct {
P []People
F string
GF string
}
rows := []*Nested{}
for i := 0; i < 10000; i++ {
row := Nested{
P: []People{
{
Age: 43,
Name: "john",
},
},
F: "testF",
GF: "testGF",
}
rows = append(rows, &row)
}
fileName := fmt.Sprintf("%s.parquet", name)
wf, err := os.OpenFile(fileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, os.ModePerm)
if err != nil {
fmt.Println("Can't create local file", err)
return
}
writer := parquet.NewGenericWriter[*Nested](wf)
writer.Write(rows)
fmt.Println("write finished")
writer.Close()
}ycyang-26 commented
hello! I meet the same question, how do you fix it?