golangci/golangci-lint

`GOCACHEPROG` implementation that store files with extensions cause `cgo preprocessing failed`

Closed this issue · 12 comments

xakep666 commented

Welcome

  • Yes, I'm using a binary release within 2 latest releases. Only such installations are supported.
  • Yes, I've searched similar issues on GitHub and didn't find any.
  • Yes, I've read the typecheck section of the FAQ.
  • Yes, I've tried with the standalone linter if available (e.g., gocritic, go vet, etc.).
  • I agree to follow this project's Code of Conduct

How did you install golangci-lint?

AUR

Description of the problem

I'm developing an internal implementation of GOCACHEPROG to store build and linter cache on S3-like storage. CACHEPROG protocol requires any file to be stored on disk. When I tried to run golangci-lint with my CACHEPROG it failed with could not import C (cgo preprocessing failed).

Digging deeper I found that packages.Load internally discards files that contains non-.go extensions. Removing extensions in CACHEPROG helper solved a problem.

To reproduce a problem I've attached a repo.

How to use it:

  1. Clone
  2. go mod download
  3. go build -o bin/diskcache ./cmd/diskcache

Run a linter in a following way:

$ GOCACHEPROG="$(pwd)/bin/diskcache --cache-dir=$(pwd)/.cache --use-extensions" golangci-lint run -v
INFO golangci-lint has version 2.2.1 built with go1.24.4 from 66496a99 on 2025-06-29T21:03:24Z 
INFO [config_reader] Config search paths: [./ <redacted> /home /] 
INFO [config_reader] Used config file .golangci.yml 
INFO [config_reader] Module name "example"        
INFO [goenv] Read go env for 3.358284ms: map[string]string{"GOCACHE":"/home/xakep666/.cache/go-build", "GOROOT":"/usr/lib/go"} 
INFO [lintersdb] Active 1 linters: [contextcheck] 
INFO [loader] Go packages loading at mode 8767 (deps|exports_file|files|imports|name|types_sizes|compiled_files) took 8.42253553s 
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 615.954µs 
INFO [linters_context/goanalysis] analyzers took 1.089482ms with top 10 stages: buildssa: 655.208µs, contextcheck: 434.274µs 
INFO [runner] Processors filtering stat (in/out): filename_unadjuster: 1/1, path_relativity: 1/1, exclusion_rules: 1/1, source_code: 1/1, sort_results: 1/1, path_absoluter: 1/1, invalid_issue: 1/1, max_same_issues: 1/1, path_shortener: 1/1, generated_file_filter: 1/1, nolint_filter: 1/1, diff: 1/1, fixer: 1/1, uniq_by_line: 1/1, max_per_file_from_linter: 1/1, severity-rules: 1/1, path_prettifier: 1/1, cgo: 1/1, exclusion_paths: 1/1, max_from_linter: 1/1 
INFO [runner] processing took 58.523µs with stages: source_code: 49.366µs, path_relativity: 2.076µs, max_same_issues: 984ns, sort_results: 826ns, uniq_by_line: 795ns, path_shortener: 589ns, nolint_filter: 542ns, max_from_linter: 528ns, filename_unadjuster: 343ns, cgo: 301ns, invalid_issue: 280ns, fixer: 274ns, exclusion_paths: 270ns, max_per_file_from_linter: 209ns, path_absoluter: 207ns, diff: 204ns, path_prettifier: 204ns, exclusion_rules: 200ns, generated_file_filter: 170ns, severity-rules: 155ns 
INFO [runner] linters took 447.523751ms with stages: goanalysis_metalinter: 447.420817ms 
../../go/pkg/mod/github.com/uber/h3-go/v4@v4.3.0/h3.go:31:8: could not import C (cgo preprocessing failed) (typecheck)
import "C"
       ^
1 issues:
* typecheck: 1
INFO File cache stats: 1 entries of total size 39.2KiB 
INFO Memory: 90 samples, avg is 37.9MB, max is 109.0MB 
INFO Execution took 8.871171776s
results in error

But run without --use-extensions finishes successfully

$ GOCACHEPROG="$(pwd)/bin/diskcache --cache-dir=$(pwd)/.cache" golangci-lint run -v
GOCACHEPROG="$(pwd)/bin/diskcache --cache-dir=$(pwd)/.cache" golangci-lint run -v                                          INT ✘ 
INFO golangci-lint has version 2.2.1 built with go1.24.4 from 66496a99 on 2025-06-29T21:03:24Z 
INFO [config_reader] Config search paths: [./ <redacted> /home /] 
INFO [config_reader] Used config file .golangci.yml 
INFO [config_reader] Module name "example"        
INFO [goenv] Read go env for 2.831168ms: map[string]string{"GOCACHE":"/home/xakep666/.cache/go-build", "GOROOT":"/usr/lib/go"} 
INFO [lintersdb] Active 1 linters: [contextcheck] 
INFO [loader] Go packages loading at mode 8767 (compiled_files|exports_file|deps|files|imports|name|types_sizes) took 8.281230895s 
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 441.503µs 
INFO [linters_context/goanalysis] analyzers took 282.207776ms with top 10 stages: buildssa: 278.607843ms, contextcheck: 3.587723ms, typecheck: 12.21µs 
INFO [runner] processing took 2.456µs with stages: exclusion_rules: 477ns, max_same_issues: 291ns, exclusion_paths: 212ns, path_absoluter: 147ns, source_code: 146ns, nolint_filter: 119ns, severity-rules: 117ns, path_relativity: 112ns, max_from_linter: 108ns, filename_unadjuster: 106ns, diff: 106ns, max_per_file_from_linter: 105ns, path_shortener: 103ns, path_prettifier: 58ns, invalid_issue: 43ns, cgo: 43ns, uniq_by_line: 41ns, fixer: 41ns, generated_file_filter: 41ns, sort_results: 40ns 
INFO [runner] linters took 723.365286ms with stages: goanalysis_metalinter: 723.324629ms 
0 issues.
INFO File cache stats: 0 entries of total size 0B 
INFO Memory: 92 samples, avg is 42.6MB, max is 142.6MB 
INFO Execution took 9.005394346

Flag --use-extensions simply adds .bin extension to file on disk.

Probably similar issue should be filed in the https://github.com/golang/go but golangci-lint was an entrypoint for this issue so I'm creating it here.

Version of golangci-lint

$ golangci-lint --version
golangci-lint has version 2.2.1 built with go1.24.4 from 66496a99 on 2025-06-29T21:03:24Z

Configuration

version: "2"
run:
  concurrency: 1
  tests: false
linters:
  default: none
  enable:
    - contextcheck

Go environment

$ go version && go env
go version go1.24.4 linux/amd64
AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/xakep666/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/xakep666/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3359120951=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/xakep666/go/pkg/mod'
GONOPROXY='gitlab.diftech.org'
GONOSUMDB='gitlab.diftech.org'
GOOS='linux'
GOPATH='/home/xakep666/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/xakep666/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.4'
GOWORK=''
PKG_CONFIG='pkg-config'

Verbose output of running

$ golangci-lint cache clean
$ golangci-lint run -v
INFO golangci-lint has version 2.2.1 built with go1.24.4 from 66496a99 on 2025-06-29T21:03:24Z 
INFO [config_reader] Config search paths: [./ <redacted> /home /] 
INFO [config_reader] Used config file .golangci.yml 
INFO [config_reader] Module name "example"        
INFO [goenv] Read go env for 2.6848ms: map[string]string{"GOCACHE":"/home/xakep666/.cache/go-build", "GOROOT":"/usr/lib/go"} 
INFO [lintersdb] Active 1 linters: [contextcheck] 
INFO [loader] Go packages loading at mode 8767 (files|imports|name|compiled_files|deps|exports_file|types_sizes) took 189.621187ms 
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 500.904µs 
INFO [linters_context/goanalysis] analyzers took 1.120949437s with top 10 stages: buildssa: 1.099012057s, contextcheck: 21.925965ms, typecheck: 11.415µs 
INFO [runner] processing took 2.546µs with stages: max_same_issues: 392ns, path_absoluter: 344ns, exclusion_paths: 316ns, exclusion_rules: 269ns, nolint_filter: 180ns, path_shortener: 108ns, max_per_file_from_linter: 105ns, source_code: 105ns, filename_unadjuster: 103ns, diff: 97ns, path_prettifier: 93ns, max_from_linter: 86ns, cgo: 50ns, path_relativity: 45ns, fixer: 44ns, invalid_issue: 44ns, generated_file_filter: 42ns, sort_results: 42ns, severity-rules: 41ns, uniq_by_line: 40ns 
INFO [runner] linters took 3.043707335s with stages: goanalysis_metalinter: 3.043677551s 
0 issues.
INFO File cache stats: 0 entries of total size 0B 
INFO Memory: 34 samples, avg is 153.2MB, max is 195.6MB 
INFO Execution took 3.234391789s

A minimal reproducible example or link to a public repository

https://github.com/xakep666/golangcilint-gocacheprog-cgo

Validation

  • Yes, I've included all information above (version, config, etc.).

Supporter

boring-cyborg commented

Hey, thank you for opening your first Issue ! 🙂 If you would like to contribute we have a guide for contributors.

ldez commented

Hello, have you tried to use this implementation: https://github.com/bradfitz/go-tool-cache/

ldez commented

The env var for the programmatic cache inside golangci-lint is GOLANGCI_LINT_CACHEPROG.

xakep666 commented

@ldez just tried bradritz's tool and got a weirder problem

$ GOCACHEPROG="$HOME/go/bin/go-cacher -cache-dir $(pwd)/.cache" golangci-lint run -v                                           1 ✘ 
INFO golangci-lint has version 2.2.1 built with go1.24.4 from 66496a99 on 2025-06-29T21:03:24Z 
INFO [config_reader] Config search paths: [./ <redacted> /home /] 
INFO [config_reader] Used config file .golangci.yml 
INFO [config_reader] Module name "example"        
INFO [goenv] Read go env for 3.280565ms: map[string]string{"GOCACHE":"/home/xakep666/.cache/go-build", "GOROOT":"/usr/lib/go"} 
INFO [lintersdb] Active 1 linters: [contextcheck] 
INFO [loader] Go packages loading at mode 8767 (compiled_files|deps|exports_file|files|imports|name|types_sizes) took 116.826412ms 
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 455.703µs 
INFO [linters_context/goanalysis] analyzers took 42.561µs with top 10 stages: typecheck: 35.794µs, buildssa: 4.27µs, contextcheck: 2.497µs 
INFO [runner] Processors filtering stat (in/out): cgo: 2/2, exclusion_paths: 2/2, nolint_filter: 2/2, diff: 2/2, severity-rules: 2/2, path_prettifier: 2/2, path_relativity: 2/2, generated_file_filter: 2/2, uniq_by_line: 2/2, max_same_issues: 2/2, source_code: 2/2, invalid_issue: 2/2, exclusion_rules: 2/2, fixer: 2/2, max_per_file_from_linter: 2/2, filename_unadjuster: 2/2, max_from_linter: 2/2, path_shortener: 2/2, sort_results: 2/2, path_absoluter: 2/2 
INFO [runner] processing took 31.756µs with stages: source_code: 19.896µs, path_relativity: 2.941µs, uniq_by_line: 1.107µs, sort_results: 1.069µs, max_same_issues: 1.068µs, path_shortener: 982ns, nolint_filter: 806ns, fixer: 572ns, max_from_linter: 542ns, path_absoluter: 349ns, cgo: 322ns, invalid_issue: 282ns, max_per_file_from_linter: 276ns, exclusion_rules: 272ns, filename_unadjuster: 267ns, path_prettifier: 262ns, exclusion_paths: 262ns, generated_file_filter: 194ns, diff: 187ns, severity-rules: 100ns 
INFO [runner] linters took 499.289113ms with stages: goanalysis_metalinter: 499.210912ms 
main.go:10:12: undefined: h3 (typecheck)
        latLng := h3.NewLatLng(37.775938728915946, -122.41795063018799)
                  ^
main.go:13:15: undefined: h3 (typecheck)
        cell, err := h3.LatLngToCell(latLng, resolution)
                     ^
2 issues:
* typecheck: 2
INFO File cache stats: 1 entries of total size 269B 
INFO Memory: 8 samples, avg is 79.9MB, max is 125.4MB 
INFO Execution took 617.114326ms

It's also familiar to me that GOLANGCI_LINT_CACHEPROG used to perform caching of golangci-lint's objects. But we found a problem by accidentally setting a GOCACHEPROG. Probably it's because golang.org/x/tools/go/packages calls go inside.

ldez commented 
$ git clone -q git@github.com:xakep666/golangcilint-gocacheprog-cgo.git
$ cd golangcilint-gocacheprog-cgo
$ go install github.com/bradfitz/go-tool-cache/cmd/go-cacher@latest
$ GOLANGCI_LINT_CACHEPROG="$(which go-cacher) --verbose" golangci-lint run 
2025/07/01 15:28:41 Defaulting to cache dir /home/ldez/.cache/go-cacher ...
2025/07/01 15:28:45 cacher: closing; 186 gets (0 hits, 186 misses, 0 errors); 188 puts (0 errors)
0 issues.
ldez commented

This is a cgo problem: it requires compilation, and so it will use the go command.

ldez commented

Note 1: In your repo example, your module name is a one-level name; those names are technically reserved for Go itself.
The namespace must be used.

Note2:

$ git clone -q git@github.com:xakep666/golangcilint-gocacheprog-cgo.git
$ cd golangcilint-gocacheprog-cgo
$ go install github.com/bradfitz/go-tool-cache/cmd/go-cacher@latest
$ GOCACHEPROG="$(which go-cacher)" go build .
2025/07/01 15:37:41 Defaulting to cache dir /home/ldez/.cache/go-cacher ...
2025/07/01 15:37:41 put(action 1c94221a561a523ae31fd0f30ff0a933f935fc9a740203ec32341c2727661ff3, obj , 4215 bytes): failed to write file to disk with right size: disk=652; wanted=4215
2025/07/01 15:37:41 put(action c0b1c86515166d812f277fa2a40cf745b40a06398a97604ed58a88c2e0840b62, obj , 652 bytes): failed to write file to disk with right size: disk=929; wanted=652
2025/07/01 15:37:41 put(action 4a925c6284608460e843d64e96e89f9e139e8d0c71ce8213be7d7abb0c1836c7, obj , 1329 bytes): failed to write file to disk with right size: disk=33104; wanted=1329
2025/07/01 15:37:41 put(action f395ac809e28bd914b65f364f8a1ce315933c507e7ed153d3876ae42a7b8727a, obj , 5344 bytes): failed to write file to disk with right size: disk=8167; wanted=5344
2025/07/01 15:37:41 put(action de8095826f92688873b59c0d3eaeb87e3d97fdc67487e61140fb206b67a55e3d, obj , 33104 bytes): failed to write file to disk with right size: disk=10358; wanted=33104
2025/07/01 15:37:41 put(action 8ce1eac908c4305b1e916343db4780472c44141dbbe20e970c39dd7c62a0788b, obj , 1702 bytes): failed to write file to disk with right size: disk=517; wanted=1702
2025/07/01 15:37:41 put(action c7eafe143a779c146c31347e41caf158ed1fa13c53085d001d4d1a8ba783e1c9, obj , 517 bytes): failed to write file to disk with right size: disk=2343; wanted=517
2025/07/01 15:37:41 put(action d8ecd5b35ce51b4277dd8cc0a6596e126ebb3dfc16a61318349d92909d5a6786, obj , 2343 bytes): failed to write file to disk with right size: disk=5551; wanted=2343
2025/07/01 15:37:41 put(action 3c354e3147b1843185eef5f0774addc32419cb8f2326e87138622f88848586c9, obj , 8167 bytes): failed to write file to disk with right size: disk=5551; wanted=8167
2025/07/01 15:37:41 put(action 3164ed2474d843737a1fa727ca95e41796dae96b1c864885e1d11cbab3c9b249, obj , 1527 bytes): failed to write file to disk with right size: disk=5265; wanted=1527
2025/07/01 15:37:41 put(action e6cc72ffa23a2dddba536b6ac71b28f98c15301317564ee51a7af82240bcd1a1, obj , 344 bytes): failed to write file to disk with right size: disk=373; wanted=344
2025/07/01 15:37:41 put(action 84a97c341abd764b1e401a06540bfb3a6c2fc54e5872a7d0b774722e838071ab, obj , 230 bytes): failed to write file to disk with right size: disk=427; wanted=230
2025/07/01 15:37:41 put(action 76c3a3853006d72b67983d0cc7c3795f11f809e0f882fba778d413c9c4349974, obj , 427 bytes): failed to write file to disk with right size: disk=255; wanted=427
2025/07/01 15:37:41 put(action 079b83eceabbbeb06562286954f582ee448e633b198e94aa95820820ea69a262, obj , 724 bytes): failed to write file to disk with right size: disk=1704; wanted=724
2025/07/01 15:37:41 put(action 6574b8f452272532b4a9f751b60b1bc97237da2e25ffa31522d3f10a561a350b, obj , 6917 bytes): failed to write file to disk with right size: disk=2533; wanted=6917
2025/07/01 15:37:41 put(action b2e4441cb33cf65f4a8d4444265b2a2665839687cf82e1c017e5e548039c5e22, obj , 6557 bytes): failed to write file to disk with right size: disk=583; wanted=6557
../../../.gvm/gos/go1.24.4/src/fmt/errors.go:8:2: package errors is not in std (/home/ldez/.gvm/gos/go1.24.4/src/errors)
../../../.gvm/gos/go1.24.4/src/internal/fmtsort/sort.go:12:2: package cmp is not in std (/home/ldez/.gvm/gos/go1.24.4/src/cmp)
../../../.gvm/gos/go1.24.4/src/internal/fmtsort/sort.go:13:2: package reflect is not in std (/home/ldez/.gvm/gos/go1.24.4/src/reflect)
../../../.gvm/gos/go1.24.4/src/internal/fmtsort/sort.go:14:2: package slices is not in std (/home/ldez/.gvm/gos/go1.24.4/src/slices)
../../../.gvm/gos/go1.24.4/src/fmt/print.go:9:2: package io is not in std (/home/ldez/.gvm/gos/go1.24.4/src/io)
../../../.gvm/gos/go1.24.4/src/fmt/print.go:10:2: package os is not in std (/home/ldez/.gvm/gos/go1.24.4/src/os)
../../../.gvm/gos/go1.24.4/src/fmt/format.go:8:2: package strconv is not in std (/home/ldez/.gvm/gos/go1.24.4/src/strconv)
../../../.gvm/gos/go1.24.4/src/sync/hashtriemap.go:10:2: package internal/sync is not in std (/home/ldez/.gvm/gos/go1.24.4/src/internal/sync)
../../../.gvm/gos/go1.24.4/src/runtime/error.go:9:2: package internal/bytealg is not in std (/home/ldez/.gvm/gos/go1.24.4/src/internal/bytealg)
../../../.gvm/gos/go1.24.4/src/runtime/covercounter.go:8:2: package internal/coverage/rtcov is not in std (/home/ldez/.gvm/gos/go1.24.4/src/internal/coverage/rtcov)
../../../.gvm/gos/go1.24.4/src/runtime/metrics.go:10:2: package internal/godebugs is not in std (/home/ldez/.gvm/gos/go1.24.4/src/internal/godebugs)
../../../.gvm/gos/go1.24.4/src/runtime/arena.go:88:2: package internal/runtime/atomic is not in std (/home/ldez/.gvm/gos/go1.24.4/src/internal/runtime/atomic)
../../../.gvm/gos/go1.24.4/src/runtime/proc.go:13:2: package internal/runtime/exithook is not in std (/home/ldez/.gvm/gos/go1.24.4/src/internal/runtime/exithook)
../../../.gvm/gos/go1.24.4/src/internal/runtime/maps/map.go:11:2: package internal/runtime/math is not in std (/home/ldez/.gvm/gos/go1.24.4/src/internal/runtime/math)
../../../.gvm/gos/go1.24.4/src/fmt/format.go:9:2: package unicode/utf8 is not in std (/home/ldez/.gvm/gos/go1.24.4/src/unicode/utf8)
xakep666 commented

Probably clarification needed here
to avoid extensions in DiskPath

type ProgResponse struct {
        // other fields

	// DiskPath is the absolute path on disk of the body corresponding to a
	// "get" (on cache hit) or "put" request's ActionID.
	DiskPath string `json:",omitempty"`
}

Leaving a comment here to help other developers of similar helpers.

  • 👍1
xakep666 commented

It's better to try go-cache-plugin by Tailscale. It is a production tool unlike bradfitz's one.

  • 👍1
ldez commented

go-cache-plugin doesn't work without an S3 bucket, so it cannot be a replacement for go-cacher for local tests.

xakep666 commented

It works like this

$ GOCACHEPROG="$HOME/go/bin/go-cache-plugin --cache-dir $(pwd)/.cache --bucket test --region auto" golangci-lint run -v

but very slow. So it's kind usable in local tests.

UPD: diskcache in my repo actually based on libraries used in go-cache-plugin but without S3. It's a little bit modified version from here.

  • 👍1
ldez commented

I had stopped the command before the end because it was taking so long, so I thought it wasn't working.