cmd/go: unable to "go get" repository with external LFS server (go modules)

Question

cmd/go: unable to "go get" repository with external LFS server (go modules)

campbellr opened this issue 5 years ago · 9 comments

What version of Go are you using (`go version`)?

$ go version
go version go1.14.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/ryancampbell/Library/Caches/go-build"
GOENV="/Users/ryancampbell/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY="github.com/kirasystems/*"
GONOSUMDB="github.com/kirasystems/*"
GOOS="darwin"
GOPATH="/Users/ryancampbell/go"
GOPRIVATE="github.com/kirasystems/*"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14.3/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/k7/np6wbqs53wvgc5167kfc81yr0000gp/T/go-build056608014=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I have a repository that use Git LFS, with a external LFS server configured via a .lfsconfig file in the root of the repository (docs).

When I attempt to go get a package in this repo in "module-aware mode", it fails with the following error:

$ go get github.com/campbellr/goproject/package
go: downloading github.com/campbellr/goproject v0.0.0-20200618225840-1c974a2e2163
go: downloading github.com/campbellr/goproject/package v0.0.0-20200618225840-1c974a2e2163
go get github.com/campbellr/goproject/package: git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 1c974a2e2163f0097eddb66edd91ebbb725cc3fc -- package in /Users/ryancampbell/go/pkg/mod/cache/vcs/bc6786f5d692de19e81d93e06b7c197cfd764a6b5981aeb750293aad946567ca: exit status 128:
        Downloading package/some-lfs-file.model (9.8 KB)
        Error downloading object: package/some-lfs-file.model (0759275): Smudge error: Error downloading package/some-lfs-file.model (0759275b587490b6e6124f421b9781362517cd1aa2354c4e849f3949adb05f94): [0759275b587490b6e6124f421b9781362517cd1aa2354c4e849f3949adb05f94] Object does not exist on the server: [404] Object does not exist on the server

        Errors logged to /Users/ryancampbell/go/pkg/mod/cache/vcs/bc6786f5d692de19e81d93e06b7c197cfd764a6b5981aeb750293aad946567ca/lfs/logs/20200619T084543.526724.log
        Use `git lfs logs last` to view the log.
        error: external filter 'git-lfs filter-process' failed
        fatal: package/some-lfs-file.model: smudge filter lfs failed

As far as I can tell, the problem seems to be due to the way that go get clones the git repository.

git init --bare
git remote add origin github.com/campbellr/goproject
git fetch
git archive thepackage/ directory

Since git archive is running in a bare repository, there is no .lfsconfig, so git-lfs doesn't fetch the objects from the right lfs server and gets a 404.

Note that the same go get will succeed in GOPATH mode (since it just does a normal git clone)

Answer 1 · 2020-06-22T15:17:04.000Z

/cc @bcmills @jayconrod @matloob

Answer 2 · 2020-06-23T15:29:21.000Z

Related #25605, especially this comment.

For posterity: I think that any repositories being built by vgo depend on a Git LFS object during compilation will fail, since those objects will not be checked out, (because the repository is bare). That's a limitation of git-archive(1) and Git LFS, but not vgo.

Answer 3 · 2020-06-23T17:37:12.000Z

tbh, I wasn't sure if this was more appropriate as a git-lfs or git-archive issue rather than a Go issue, but I ended up filing it here because:

This is a regression from GOPATH mode go get, which works as expected
This is a result of go get cloning the repo in an "unexpected" way. I'm sure there is a good reason for it, but it does seem break assumptions that git-lfs (and possibly other tools?) seems to depend on.

Answer 4 · 2020-06-23T20:16:50.000Z

See also #29987 (also related to LFS).

Answer 5 · 2020-06-23T20:18:22.000Z

See also #38941 (comment) (somehow related to git-lfs smudging).

Answer 6 · 2024-08-13T19:30:47.000Z

This issue is four years old. Why isn't this fixed yet?

Answer 7 · 2024-08-13T19:55:24.000Z

@pebbe This is open source software and this is not a common case. If you are running into this I would encourage you to send in a patch. See https://go.dev/doc/contribute. Thanks.

Answer 8 · 2024-08-13T21:26:40.000Z

@pebbe This is open source software and this is not a common case. If you are running into this I would encourage you to send in a patch. See https://go.dev/doc/contribute. Thanks.

I don't have the expertise to fix go get.

Answer 9 · 2024-08-13T21:33:24.000Z

This is weird. I have this file main.go:

package main

import (
    "fmt"
    "github.com/pebbe/tokenize"
)

func main() {
    s, err := tokenize.Dutch("Dit is een test. En dit, misschien?", true)
    fmt.Println(err)
    fmt.Println(s)
}

I try to build this (Go version 1.23.0), but it fails because of lfs:

$ go mod init local
go: creating new go.mod: module local
go: to add module requirements and sums:
        go mod tidy

$ go mod tidy
go: finding module for package github.com/pebbe/tokenize
go: downloading github.com/pebbe/tokenize v1.0.4
go: found github.com/pebbe/tokenize in github.com/pebbe/tokenize v1.0.4

$ go build .
# github.com/pebbe/tokenize
libtok1.c:1:1: error: unknown type name ‘version’
    1 | version https://git-lfs.github.com/spec/v1
      | ^~~~~~~
libtok1.c:1:14: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘:’ token
    1 | version https://git-lfs.github.com/spec/v1
      |              ^
libtok1.c:2:12: error: invalid suffix "c540c13c58219a3aac14ed2bf7c1b6396664268f3abc0f8f94d8374b889882" on integer constant
    2 | oid sha256:90c540c13c58219a3aac14ed2bf7c1b6396664268f3abc0f8f94d8374b889882
      |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Let's begin again:

$ go clean -cache -modcache
$ rm go.*

I set this variable, but I don't know what this is suppose to change:

$ export GOPRIVATE=github.com/pebbe/tokenize

Now try again. It works. I notice that go mod tidy now takes much more time to complete.

$ export GOPRIVATE=github.com/pebbe/tokenize

$ go mod init local
go: creating new go.mod: module local
go: to add module requirements and sums:
        go mod tidy

$ go mod tidy
go: finding module for package github.com/pebbe/tokenize
go: downloading github.com/pebbe/tokenize v1.0.4
go: found github.com/pebbe/tokenize in github.com/pebbe/tokenize v1.0.4

$ go build .

$ ./local 
<nil>
Dit is een test .
En dit , misschien ?

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?