k8sgpt-ai/k8sgpt

[Bug]: Trivy Analyzer has concurrent map write issue

Opened this issue · 2 comments

michael12312 commented

Checklist

  • I've searched for similar issues and couldn't find anything matching
  • I've included steps to reproduce the behavior

Affected Components

  • K8sGPT (CLI)
  • K8sGPT Operator

K8sGPT Version

No response

Kubernetes Version

No response

Host OS and its Version

No response

Steps to reproduce

  1. Enable Trivy
  2. Install a KIND Cluster and enabled argocd
  3. Run alanlyzer

k8sgpt integrations activate trivy
k8sgpt analyze

Then it failed with below error:

fatal error: concurrent map writes
fatal error: concurrent map writes

goroutine 48 [running]:

k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypeWithName(0xc00056dd50, {{0x10a763d15, 0x16}, {0x10a7379e5, 0x8}, {0x10aff49c3, 0x1c}}, {0x10b91fcd0, 0xc0005741c0})
/Users/test/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/runtime/scheme.go:174 +0x2eb
k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypes(0xc00056dd50, {{0x10a763d15, 0x16}, {0x10a7379e5, 0x8}}, {0xc000eb3b40, 0x16, 0x16})
/Users/test/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/runtime/scheme.go:148 +0x2af
github.com/aquasecurity/trivy-operator/pkg/apis/aquasecurity/v1alpha1.addKnownTypes(0xc00056dd50)
/Users/test/go/pkg/mod/github.com/aquasecurity/trivy-operator@v0.17.1/pkg/apis/aquasecurity/v1alpha1/register.go:22 +0x12b0
k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(0x10d32c040, 0xc00056dd50)
/Users/test/go/pkg/mod/k8s.io/apimachinery@v0.30.2/pkg/runtime/scheme_builder.go:29 +0x82
github.com/k8sgpt-ai/k8sgpt/pkg/integration/trivy.TrivyAnalyzer.analyzeVulnerabilityReports({0x1, 0x0}, {0xc00087d440, {0x10b9365d0, 0x10d3b4d88}, {0x0, 0x0}, {0x0, 0x0}, 0x0, ...})
/Users/test/A10/k8sgpt/pkg/integration/trivy/analyzer.go:43 +0x102
github.com/k8sgpt-ai/k8sgpt/pkg/integration/trivy.TrivyAnalyzer.Analyze({0x1, 0x0}, {0xc00087d440, {0x10b9365d0, 0x10d3b4d88}, {0x0, 0x0}, {0x0, 0x0}, 0x0, ...})
/Users/test/A10/k8sgpt/pkg/integration/trivy/analyzer.go:161 +0xbe
github.com/k8sgpt-ai/k8sgpt/pkg/analysis.(*Analysis).RunAnalysis.func3({0x10b903e80, 0xc000d2721a}, {0xc001494060, 0x13})

/Users/test/A10/k8sgpt/pkg/analysis/analysis.go:271 +0x155

created by github.com/k8sgpt-ai/k8sgpt/pkg/analysis.(*Analysis).RunAnalysis in goroutine 1
/Users/test/A10/k8sgpt/pkg/analysis/analysis.go:269 +0xb74

Expected behaviour

no concurrent map write fatal error.

Actual behaviour

fatal error

Additional Information

No response

michael12312 commented

hi, I will fix this, and create PR soon

  • 👍2
rohitg00 commented

I have implemented a solution and I'm ready to submit a PR if you approve.

  1. Proposed Solution:
    To address this issue, we propose introducing a mutex lock to synchronize access to the mergedAnalyzerMap. This will ensure that only one goroutine can modify the map at a time, preventing race conditions and maintaining data integrity.

  2. Implementation Steps:
    a. Import the sync package in the relevant files.
    b. Add a sync.Mutex field to the Trivy struct in the trivy.go file.
    c. Modify the AddAnalyzer method to use the mutex for synchronization:

    • Acquire the lock before modifying the mergedAnalyzerMap.
    • Release the lock after the modification is complete.
      d. Update any other methods that access or modify the mergedAnalyzerMap to use the mutex.

  3. Impact Analysis:

    • Positive impacts:
      • Eliminates race conditions and ensures data integrity.
      • Improves overall stability and reliability of the Trivy Analyzer integration.
    • Potential drawbacks:
      • Slight performance overhead due to mutex operations.
      • May introduce potential deadlocks if not implemented carefully.

  4. Testing Plan:
    a. Unit Tests:

    • Develop unit tests that simulate concurrent access to the AddAnalyzer method.
    • Verify that the mutex correctly synchronizes access to the mergedAnalyzerMap.
      b. Integration Tests:
    • Create integration tests that exercise the Trivy Analyzer in a concurrent environment.
    • Ensure that no race conditions occur during parallel execution of multiple analyzers.
      c. Performance Testing:
    • Conduct performance tests to measure the impact of the mutex on overall system performance.
    • Compare results with the previous implementation to ensure acceptable performance.
      d. Code Review:
    • Perform a thorough code review to ensure proper implementation of the mutex and identify any potential issues.

  5. Implementation Details:
    File: /home/ubuntu/k8sgpt/pkg/integration/trivy/trivy.go

    • Add a new field to the Trivy struct: mu sync.Mutex
    • Modify the AddAnalyzer method: 
      func (t *Trivy) AddAnalyzer(mergedMap *map[string]common.IAnalyzer) {
    t.mu.Lock()
    defer t.mu.Unlock()
    // Existing code for adding analyzers
}

    File: /home/ubuntu/k8sgpt/pkg/analyzer/analyzer.go

    • Ensure that the call to AddAnalyzer is thread-safe: 
      in.AddAnalyzer(&mergedAnalyzerMap)
      (This call is already correct, as the synchronization is handled within the AddAnalyzer method)