Out of memory crash when trying to scan repo with lots of files

[robinsmidsrod]

What happened?

I tried to scan a git repo that contains a lot of files, most of them binary and I got an out of memory error that caused the application to crash.

What did you expect to happen?

For the scan to complete without an out of memory issue.

How can we reproduce this?

Try to run pluto on a directory with many files. In my case the directory had 147k files, most of them small MP3 sound files (don't ask why they're in a source code repo, it's a whole other story).

Version

5.11.1

Search

• I did search for other open and closed issues before opening this.

Code of Conduct

• I agree to follow this project's Code of Conduct

Additional context

Command I ran: "$PLUTO" detect-files -d "$REPO_PATH" --target-versions "k8s=v${K8S_VERSION}"

Stack-trace from error:

Running pluto 5.11.1 scan for deprecated Kubernetes 1.24.0 apiVersions...
fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x18bc30a?, 0x400000?})
        /usr/local/go/src/runtime/panic.go:992 +0x71
runtime.sysMap(0xc0d1800000, 0xc000079e90?, 0xc000079ef8?)
        /usr/local/go/src/runtime/mem_linux.go:189 +0x11b
runtime.(*mheap).grow(0x26edae0, 0x3?)
        /usr/local/go/src/runtime/mheap.go:1404 +0x225
runtime.(*mheap).allocSpan(0x26edae0, 0x3, 0x0, 0x71)
        /usr/local/go/src/runtime/mheap.go:1170 +0x171
runtime.(*mheap).alloc.func1()
        /usr/local/go/src/runtime/mheap.go:912 +0x65
runtime.systemstack()
        /usr/local/go/src/runtime/asm_amd64.s:469 +0x49

goroutine 1 [running]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:436 fp=0xc0005297f0 sp=0xc0005297e8 pc=0x461180
runtime.(*mheap).alloc(0x203034?, 0x203034?, 0xa0?)
        /usr/local/go/src/runtime/mheap.go:906 +0x65 fp=0xc000529838 sp=0xc0005297f0 pc=0x426745
runtime.(*mcentral).grow(0x6000?)
        /usr/local/go/src/runtime/mcentral.go:244 +0x5b fp=0xc000529880 sp=0xc000529838 pc=0x41757b
runtime.(*mcentral).cacheSpan(0x27031e0)
        /usr/local/go/src/runtime/mcentral.go:164 +0x30f fp=0xc0005298d8 sp=0xc000529880 pc=0x4173af
runtime.(*mcache).refill(0x7f0a5cc0a108, 0x71)
        /usr/local/go/src/runtime/mcache.go:162 +0xaf fp=0xc000529910 sp=0xc0005298d8 pc=0x416a2f
runtime.(*mcache).nextFree(0x7f0a5cc0a108, 0x71)
        /usr/local/go/src/runtime/malloc.go:886 +0x85 fp=0xc000529958 sp=0xc000529910 pc=0x40cac5
runtime.mallocgc(0x2c15, 0x15f9480, 0x1)
        /usr/local/go/src/runtime/malloc.go:1085 +0x4e5 fp=0xc0005299d0 sp=0xc000529958 pc=0x40d145
runtime.makeslice(0xc0d13daa30?, 0x2c?, 0x0?)
        /usr/local/go/src/runtime/slice.go:103 +0x52 fp=0xc0005299f8 sp=0xc0005299d0 pc=0x44a3b2
os.ReadFile({0xc0004db110?, 0x7f0a5cc0a108?})
        /usr/local/go/src/os/file.go:693 +0xeb fp=0xc000529ad8 sp=0xc0005299f8 pc=0x4d3eab
io/ioutil.ReadFile(...)
        /usr/local/go/src/io/ioutil/ioutil.go:37
github.com/fairwindsops/pluto/v5/pkg/finder.(*Dir).CheckForAPIVersion(0xc000122780, {0xc0004db110, 0x2c})
        /home/circleci/go/src/github.com/fairwindsops/pluto/pkg/finder/finder.go:125 +0x35 fp=0xc000529b48 sp=0xc000529ad8 pc=0x11b70d5
github.com/fairwindsops/pluto/v5/pkg/finder.(*Dir).scanFiles(0xc000122780)
        /home/circleci/go/src/github.com/fairwindsops/pluto/pkg/finder/finder.go:110 +0x12f fp=0xc000529c18 sp=0xc000529b48 pc=0x11b6e2f
github.com/fairwindsops/pluto/v5/pkg/finder.(*Dir).FindVersions(0xc000122780)
        /home/circleci/go/src/github.com/fairwindsops/pluto/pkg/finder/finder.go:73 +0x37 fp=0xc000529c98 sp=0xc000529c18 pc=0x11b68d7
github.com/fairwindsops/pluto/v5/cmd.glob..func3(0x26bac80?, {0x18a865f?, 0x4?, 0x4?})
        /home/circleci/go/src/github.com/fairwindsops/pluto/cmd/root.go:283 +0x4a fp=0xc000529d40 sp=0xc000529c98 pc=0x151340a
github.com/spf13/cobra.(*Command).execute(0x26bac80, {0xc000391700, 0x4, 0x4})
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876 +0x67b fp=0xc000529e18 sp=0xc000529d40 pc=0x14a07fb
github.com/spf13/cobra.(*Command).ExecuteC(0x26baa00)
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x3b4 fp=0xc000529ed0 sp=0xc000529e18 pc=0x14a0e94
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
github.com/fairwindsops/pluto/v5/cmd.Execute({0x1aec700?, 0x60?}, {0x1af97c0?, 0x0?}, {0x263c0a0?, 0x25ed488?, 0xc0000021a0?})
        /home/circleci/go/src/github.com/fairwindsops/pluto/cmd/root.go:434 +0xb2 fp=0xc000529f38 sp=0xc000529ed0 pc=0x1514272
main.main()
        /home/circleci/go/src/github.com/fairwindsops/pluto/cmd/pluto/main.go:31 +0x4a fp=0xc000529f80 sp=0xc000529f38 pc=0x15144ea
runtime.main()
        /usr/local/go/src/runtime/proc.go:250 +0x212 fp=0xc000529fe0 sp=0xc000529f80 pc=0x436672
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000529fe8 sp=0xc000529fe0 pc=0x463261

goroutine 7 [sleep]:
time.Sleep(0x6fc23ac00)
        /usr/local/go/src/runtime/time.go:194 +0x12e
sigs.k8s.io/controller-runtime/pkg/log.init.0.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/log/log.go:63 +0x38
created by sigs.k8s.io/controller-runtime/pkg/log.init.0
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/log/log.go:62 +0x25

[sudermanjr]

Thanks for reporting! It seems like we could fairly easily modify this section of the code to either exclude files that don't end in yml or yaml, or even better look at the file type to exclude non yaml files (using something like github.com/h2non/filetype).

We might be able to get this in, but it would also make a really good first issue if it's something you wanted to contribute.

[robinsmidsrod]

I have no experience with Go, unfortunately, so it would be hard to work on.