bigcsvreader: thread #11 could not read line at offset 7465238018 (bufio: buffer full)
Closed this issue · 2 comments
madhur commented
Description of the bug
I am trying to read the 12 GB csv file with 63 million rows.
Getting an error while reading file
bigcsvreader: thread #11 could not read line at offset 7465238018 (bufio: buffer full)
Expected behavior
It is able to read the file entirely
Minimal, Complete, and Reproducible Example
Additional context
➜ go version [☸ eks-pt (mec)]
go version go1.20.4 linux/amd64
~/Desktop/python master [!][ v20.2.0][🐍 v3.11.3]
➜ go env [☸ eks-pt (mec)]
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/madhur/.cache/go-build"
GOENV="/home/madhur/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/madhur/.cache/go-cache"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/madhur/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/madhur/Desktop/python/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3011484936=/tmp/go-build -gno-record-gcc-switches"
Thank you for taking the time to report this bug!
By submitting it, you consent to comply with the Code of Conduct.
bogcon commented
Hi @madhur
You have to increase the value of CsvReader.BufferSize setting.