Error: concurrent map writes
darrenwhighamfd opened this issue · 10 comments
darrenwhighamfd commented
What happened?
When trying to run nova find on my eks cluster it fails with the following error:
fatal error: concurrent map writes
goroutine 154 [running]:
runtime.throw({0x2444bc8, 0x10})
/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0044d7e60 sp=0xc0044d7e30 pc=0x1033211
runtime.mapassign_faststr(0x6563727500000003, 0xc00a9e9710, {0xc00a9e9710, 0xf})
/usr/local/go/src/runtime/map_faststr.go:211 +0x39c fp=0xc0044d7ec8 sp=0xc0044d7e60 pc=0x1012a7c
github.com/fairwindsops/nova/pkg/helm.(*ArtifactHubPackageClient).MultiSearch.func1(0x747365742f737473, {0xc00a9e9710, 0xf}, 0xc000340060, 0xc007a0de88)
/go/src/github.com/fairwindsops/nova/pkg/helm/artifacthub.go:264 +0x196 fp=0xc0044d7fa8 sp=0xc0044d7ec8 pc=0x20464f6
github.com/fairwindsops/nova/pkg/helm.(*ArtifactHubPackageClient).MultiSearch·dwrap·3()
/go/src/github.com/fairwindsops/nova/pkg/helm/artifacthub.go:273 +0x3e fp=0xc0044d7fe0 sp=0xc0044d7fa8 pc=0x204631e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0044d7fe8 sp=0xc0044d7fe0 pc=0x1063041
created by github.com/fairwindsops/nova/pkg/helm.(*ArtifactHubPackageClient).MultiSearch
/go/src/github.com/fairwindsops/nova/pkg/helm/artifacthub.go:259 +0x605
EKS Version: 1.21
What did you expect to happen?
Standard output of Nova find
How can we reproduce this?
Re run Nova version commands.
Version
3.0.2
Search
- I did search for other open and closed issues before opening this.
Code of Conduct
- I agree to follow this project's Code of Conduct
Additional context
No response
sudermanjr commented
Did something interrupt the process by chance? This is an odd error. Unless we can find a way to reproduce, this will be difficult to triage.
darrenwhighamfd commented
Nothing I can see, This seems to happen every-time I run Nova find. Run from my local machine to an EKS cluster. I'm able to connect fine to the cluster and run other tools such as Pluto with no errors.
sudermanjr commented
Just to be certain, can you try the latest version of Nova? I'm unable to reproduce this in EKS with either 3.0.2 or 3.1.0, but it's worth a shot.
markandersontrocme commented
@sudermanjr I've also just started running into this issue.
Got it when I cloned the repo locally and built it, so it's the latest master. Happens on the master branch and with the current changes I am doing.
For the version I have installed
$ nova version
Version:3.0.2 Commit:f8f0448ca2362fff6797578a595308dd956ce462
Seems inconsistent as I can re run it and get the proper output.
$ ./nova find
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
[proper output]
$ ./nova find --show-old
Release Name Installed Latest Old Deprecated
============ ========= ====== === ==========
[proper output]
$ ./nova find --show-old
fatal error: concurrent map writes
goroutine 12 [running]:
runtime.throw({0x24c9683?, 0x10?})
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/panic.go:992 +0x71 fp=0xc002715e70 sp=0xc002715e40 pc=0x1034ad1
runtime.mapassign_faststr(0x3?, 0xc0005e42a0?, {0xc0005e42a0, 0x16})
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/map_faststr.go:212 +0x39c fp=0xc002715ed8 sp=0xc002715e70 pc=0x101485c
github.com/fairwindsops/nova/pkg/helm.(*ArtifactHubPackageClient).MultiSearch.func1(0x0?, {0xc0005e42a0, 0x16}, 0xc00078c018, 0xc002562a08)
/Users/mark.anderson/alti/nova/pkg/helm/artifacthub.go:264 +0x18c fp=0xc002715fa8 sp=0xc002715ed8 pc=0x209672c
github.com/fairwindsops/nova/pkg/helm.(*ArtifactHubPackageClient).MultiSearch.func2()
/Users/mark.anderson/alti/nova/pkg/helm/artifacthub.go:273 +0x3b fp=0xc002715fe0 sp=0xc002715fa8 pc=0x209657b
runtime.goexit()
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc002715fe8 sp=0xc002715fe0 pc=0x1064e81
created by github.com/fairwindsops/nova/pkg/helm.(*ArtifactHubPackageClient).MultiSearch
/Users/mark.anderson/alti/nova/pkg/helm/artifacthub.go:259 +0x63b
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc0005dab00?)
/usr/local/Cellar/go/1.18.2/libexec/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xa?)
/usr/local/Cellar/go/1.18.2/libexec/src/sync/waitgroup.go:136 +0x52
github.com/fairwindsops/nova/pkg/helm.(*ArtifactHubPackageClient).MultiSearch(0xc000334150, {0xc0021d9680, 0x23, 0xc000503d48?})
/Users/mark.anderson/alti/nova/pkg/helm/artifacthub.go:275 +0x309
github.com/fairwindsops/nova/cmd.glob..func2(0x329a0e0?, {0x24b7630?, 0x1?, 0x1?})
/Users/mark.anderson/alti/nova/cmd/root.go:276 +0x102d
github.com/spf13/cobra.(*Command).execute(0x329a0e0, {0xc000119470, 0x1, 0x1})
/Users/mark.anderson/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:860 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0x3299e60)
/Users/mark.anderson/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
/Users/mark.anderson/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
github.com/fairwindsops/nova/cmd.Execute({0x26fcba0?, 0x0?}, {0x27099c0?, 0xc0000021a0?})
/Users/mark.anderson/alti/nova/cmd/root.go:328 +0x7e
main.main()
/Users/mark.anderson/alti/nova/main.go:29 +0x35
...
I also got this error:
$ nova find
F0520 09:53:52.197427 10332 root.go:278] Error getting artifacthub package repos: failed to search for packages for term rbac-manager
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0x1)
/Users/mark.anderson/go/pkg/mod/k8s.io/klog/v2@v2.60.1/klog.go:860 +0x8a
k8s.io/klog/v2.(*loggingT).output(0x32b23c0, 0x3, 0x0, 0xc001d22e70, 0x1, {0x2aa8433?, 0x1?}, 0xc000580000?, 0x0)
/Users/mark.anderson/go/pkg/mod/k8s.io/klog/v2@v2.60.1/klog.go:825 +0x686
k8s.io/klog/v2.(*loggingT).printfDepth(0x32b23c0, 0x192798?, 0x0, {0x0, 0x0}, 0x0?, {0x24e9f44, 0x2b}, {0xc001a7fe60, 0x1, ...})
/Users/mark.anderson/go/pkg/mod/k8s.io/klog/v2@v2.60.1/klog.go:630 +0x1f2
k8s.io/klog/v2.(*loggingT).printf(...)
/Users/mark.anderson/go/pkg/mod/k8s.io/klog/v2@v2.60.1/klog.go:612
k8s.io/klog/v2.Fatalf(...)
/Users/mark.anderson/go/pkg/mod/k8s.io/klog/v2@v2.60.1/klog.go:1516
github.com/fairwindsops/nova/cmd.glob..func2(0x329a0e0?, {0x24b7630?, 0x2?, 0x2?})
/Users/mark.anderson/alti/nova/cmd/root.go:278 +0x10fa
github.com/spf13/cobra.(*Command).execute(0x329a0e0, {0xc0003f8860, 0x2, 0x2})
/Users/mark.anderson/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:860 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0x3299e60)
/Users/mark.anderson/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
/Users/mark.anderson/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
github.com/fairwindsops/nova/cmd.Execute({0x26fcba0?, 0x0?}, {0x27099c0?, 0xc0000021a0?})
/Users/mark.anderson/alti/nova/cmd/root.go:328 +0x7e
main.main()
/Users/mark.anderson/alti/nova/main.go:29 +0x35
sudermanjr commented
I'm a bit confused, as 3.0.2 is not the latest, and the --show-old flag doesn't exist. I assume that's what you're working on.
If someone can provide more exact steps to reproduce with 3.1.0 (preferably the version built in our releases), I would greatly appreciate the help. I'm still unable to re-create the issue.
markandersontrocme commented
@sudermanjr I just update nova and was able to reproduce it:
$ nova version
Version:3.1.0 Commit:ff8ed043ae81ed8b09e92314f1ded1b7926a306d
$ nova find
fatal error: concurrent map read and map write
goroutine 65 [running]:
runtime.throw({0x2495074, 0x0})
/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0006b3e68 sp=0xc0006b3e38 pc=0x1033391
runtime.mapaccess2_faststr(0x0, 0xc001a4e940, {0xc001a4e940, 0xe})
/usr/local/go/src/runtime/map_faststr.go:116 +0x3d4 fp=0xc0006b3ed0 sp=0xc0006b3e68 pc=0x1012694
github.com/fairwindsops/nova/pkg/helm.(*ArtifactHubPackageClient).MultiSearch.func1(0x0, {0xc001a4e940, 0xe}, 0xc000582020, 0xc0024bc428)
/go/src/github.com/fairwindsops/nova/pkg/helm/artifacthub.go:267 +0x211 fp=0xc0006b3fa8 sp=0xc0006b3ed0 pc=0x2077ff1
github.com/fairwindsops/nova/pkg/helm.(*ArtifactHubPackageClient).MultiSearch·dwrap·3()
/go/src/github.com/fairwindsops/nova/pkg/helm/artifacthub.go:273 +0x3e fp=0xc0006b3fe0 sp=0xc0006b3fa8 pc=0x2077d9e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0006b3fe8 sp=0xc0006b3fe0 pc=0x1063b81
created by github.com/fairwindsops/nova/pkg/helm.(*ArtifactHubPackageClient).MultiSearch
/go/src/github.com/fairwindsops/nova/pkg/helm/artifacthub.go:259 +0x5e5
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x0)
/usr/local/go/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0xc00000000a)
/usr/local/go/src/sync/waitgroup.go:130 +0x71
github.com/fairwindsops/nova/pkg/helm.(*ArtifactHubPackageClient).MultiSearch(0xc0003e4240, {0xc001aff8c0, 0x23, 0x23})
/go/src/github.com/fairwindsops/nova/pkg/helm/artifacthub.go:275 +0x2a5
github.com/fairwindsops/nova/cmd.glob..func2(0x3329140, {0x2472888, 0x0, 0x0})
/go/src/github.com/fairwindsops/nova/cmd/root.go:270 +0xd5c
github.com/spf13/cobra.(*Command).execute(0x3329140, {0x3371088, 0x0, 0x0})
/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:860 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0x3328ec0)
/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
github.com/fairwindsops/nova/cmd.Execute({0x267349c, 0x1018a65}, {0x2699d20, 0xc0000001a0})
/go/src/github.com/fairwindsops/nova/cmd/root.go:322 +0x7e
main.main()
/go/src/github.com/fairwindsops/nova/main.go:29 +0x35
goroutine 33 [sleep]:
time.Sleep(0x6fc23ac00)
/usr/local/go/src/runtime/time.go:193 +0x12e
sigs.k8s.io/controller-runtime/pkg/log.init.0.func1()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/log/log.go:63 +0x38
created by sigs.k8s.io/controller-runtime/pkg/log.init.0
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/log/log.go:62 +0x25
...
If you run nova find a few times in a row you will get the error at some point (I ran it 5 times).
sudermanjr commented
@markandersontrocme would you be able to try #115 and see if that takes care of the issue?
markandersontrocme commented
@sudermanjr Didn't get the concurrent map write issue but one related to artifacthub: #115 (comment)
sudermanjr commented
Thanks. I think the artifacthub issue is just bad logging. We need to remove the Fatal error, and just log the error. I'll look into it though.
sudermanjr commented
Related to logging: #116