cmd/go: remove -buildmode=shared (not c-shared)

Question

cmd/go: remove -buildmode=shared (not c-shared)

rsc opened this issue 4 years ago · 66 comments

-buildmode=shared has been broken since modules, and it is apparently unused.
See #47257 (comment).
Let's remove it.

Answer 1 · 2021-08-18T18:13:20.000Z

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Answer 2 · 2021-08-19T07:48:35.000Z

I really want to use this buildmode=shared mode.

Answer 3 · 2021-08-19T10:28:59.000Z

@rsc how has it been broken since modules?

Answer 4 · 2021-08-19T13:34:23.000Z

Just to clarify, this is not related to --buildmode=plugin, right?

Answer 5 · 2021-08-19T19:15:48.000Z

@electricface Why do you want to use it? Are you using it today?

@scott-cotton You can't use -buildmode=shared with a module and then use -linkshared to link against it from other modules. The whole idea of -buildmode=shared doesn't fit very well with modules. Each separate program can be linked against different versions of dependent modules, so if it did work people would in practice wind up with many different versions of the shared libraries of the same modules.

@ainar-g Correct, this is not about -buildmode=plugin, although plugins have their own, different, problems.

Answer 6 · 2021-08-20T02:39:15.000Z

@ianlancetaylor
I am a debian-baseed Linux Desktop Environment developer. Our team have developed lot of daemon programs with golang. But these programs have significantly larger binary file size and use more ram than C or C++ programs. We currently working on these two problems. An idea is making some modules pluggable, which means only load modules that we need.

As other C or C++ programs are using shared-library, we are researching on some shared-library-like mechanism of golang. Inspired by buildmode=plugin and goloader, we found a way that bases on buildmode=shared to make modules pluggable recently, you can found a demo here. This solution has not been using in production environment, because of some problems hasn't been solved yet. I found out that programs built with -linkeshared can not be debugged using gdb and dlv.

After this commit, our "unoffical plugin" solution can work fine with go 1.14.9~1.16.7. Programs built in this way can run stably. But this won't work on go 1.17. Program will exit with a "fatal error: unreachable method called. linker bug?"

Plan of remove buildmode=shared make me feel bad.

Answer 7 · 2021-08-20T19:16:51.000Z

@electricface It sounds like you want plugins, which are supported (though not very well) by -buildmode=plugin. We aren't going to support -buildmode=shared just to support plugins, since we already have -buildmode=plugin.

Answer 8 · 2021-08-21T01:52:04.000Z

@ianlancetaylor
I know that using static linking can eliminate the problems caused by the mismatched version of the dependent dynamic library.
But in the linux desktop system ecosystem, C or C++ languages are mostly used. These languages generally use dynamic libraries to save memory. The main reason for not using -buildmode=plugin is that static linking cannot save memory. For example, if the modules are divided into smaller ones and a lot of modules are added, each module occupies a relatively large amount of memory, and the overall occupancy will eventually become unacceptably large. Compared with C or C++ languages, they only use less memory to implement these functions, while the golang implementation uses several times the memory.
There is even an idea in our team that we must change the development language if we can't reduce the memory. I personally like to use golang for development, so I plan to study this part of the technology.

Answer 9 · 2021-08-25T17:29:17.000Z

"I want to use it" is different from "I am using it today".
We believe that it is not possible to use it today, and furthermore it is difficult to provide.
We shouldn't advertise something that doesn't actually work.

Answer 10 · 2021-08-25T18:26:30.000Z

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

Answer 11 · 2021-08-25T19:37:10.000Z

Just to confirm, this proposal suggests to remove --buildmode=shared while keeping --buildmode=c-shared? I am using --buildmode=c-shared for a number of projects and would be very sad to see it go.

Answer 12 · 2021-08-25T23:00:43.000Z

This proposal is only about -buildmode=shared.

Answer 13 · 2021-09-01T18:02:53.000Z

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

Answer 14 · 2021-09-02T07:38:00.000Z

@rsc not sure if this is a good place to give feedback about the proposal process, but this is the first time (since the proposal process exists) that I missed an issue. I was mildly interested in this (not enough now to vehemently protest), but even if I'm subscribed to the proposal process issue and I normally read minutes every week, I missed this. I took 10 days off for vacation, and those 10 days covered the only 2 emails where this issue was mentioned, before today when it was approved. Again, I don't mind about this specific issue at this point but I wanted to raise a general point.

Maybe a proposal should stay a minimum time in the proposal process before being closed, so that even people that don't work on Go full time have time to notice and react even with personal life happening in-between.

Answer 15 · 2021-09-09T04:05:26.000Z

To limit binary file size, I use command go install -buildmode=shared std to install libstd.so. And I build my application by go build -linkshared. I have been using this in production environment for a long time. I hope I can still use it in the furture.
Can we keep -buildmode=shared std, it's useful.

Answer 16 · 2021-09-09T04:21:47.000Z

Our company has been using the shared mode to reduce the binary size for a long time. The plugin mode is not useful in some scenarios cause the entire runtime package needs to be compiled and packed. So we strongly hope that shared mode can be reserved.

Answer 17 · 2021-10-22T17:50:18.000Z

We also use buildmode=shared to reduce binary size / allow pages to be shared across several binaries using a common set of packages when running in on devices with (relatively) low RAM. Hopefully a solution can be found to make this viable with modules, but I'd prefer for the option to not disappear. Without this, we would have to resort to a busybox like binary which is not ideal as each binary is maintained by a different set of people and released independently.

Answer 18 · 2021-10-27T19:20:09.000Z

Change https://golang.org/cl/359096 mentions this issue: cmd/go: remove support for -buildmode=shared

Answer 19 · 2021-10-27T21:41:34.000Z

It's very strange decision to remove shared completely. I think shared or plugin mode was not well designed and now, we can't use golang with something like java jars.

Answer 20 · 2021-10-27T21:44:24.000Z

@grolfcry Note that go build -buildmode=c-shared still works. I don't see how -buildmode=shared is related to Java jars.

Answer 21 · 2021-10-27T21:52:47.000Z

How I can build go dependencies as external libs now if I cant build shared libs and link shared? Java jars relate go shared libs and java fat jar like default go build (one executable).

Answer 22 · 2021-10-27T21:59:59.000Z

So, just to be clear, you are mentioning Java jars as an analogy? You're not literally trying to use Go with Java jars?

Yes, this is functionality that will no longer be available. But it already didn't work for anything other than the standard library. Perhaps we can introduce some similar functionality in a way that can actually work, but it won't be go build -buildmode=shared.

Answer 23 · 2021-10-27T22:10:02.000Z

Of course, java jars it's analogy.
I can use any go module (any lib) as external dependency with shared mode and link shared, not only for standard modules in golang earlier 1.16 - #47455 (comment).
Now it's impossible, and it's very strange way of language evolution - If we can't do it right, we won't do it at all.

Answer 24 · 2021-10-27T22:20:22.000Z

Yes: it used to work, but it wasn't possible to make it work with modules.

My apologies for the loss of functionality.

Answer 25 · 2021-10-29T01:22:09.000Z

Change https://golang.org/cl/359575 mentions this issue: Revert "cmd/go: remove support for -buildmode=shared"

Answer 26 · 2021-10-29T16:43:08.000Z

@ianlancetaylor, @bcmills, and I talked a bit more about this yesterday.
I had not seen the comments from Sept 2 onward until the last couple days.
(I can't keep up with all the GitHub mail I get,
and once a proposal is accepted I tend not to watch it.)

We've decided to leave -buildmode=shared alone (not delete it) for Go 1.18,
but it's not going to work any better than it does today.
In particular, we're not fixing any bugs in it anymore,
and we still intend to remove it in a future release.
However, I would like to better understand how to serve existing use cases, if that's possible.

The most fundamental bug is that -buildmode=shared stores state in $GOROOT/pkg (#22196),
making that directory not a cache that can be deleted and recreated at any time.
This made a little bit of sense when you had to go install individual packages
in order to use them in future builds, but not really, and especially not now that
go install is completely unnecessary for packages.

A second important bug is that nothing about the shared objects is cached properly (#24034).
Consider this pair of commands:

go install -buildmode=shared std
go build -linkshared helloworld.go

The go build repeats all the work of the first go install,
rebuilding the entire standard library in shared mode and creating a new libstd.so,
installing it over the (identical) existing one.
If you run the go build again, it does all the work yet again.
Every time.

This makes builds using -linkshared incredibly slow,
and it also means that they don't work at all without write access to $GOROOT.

A third important bug is that the toolchain fundamentally assumes that packages
are being compiled against the exact versions of the packages they import (#19389, #21510).
For example, suppose draw.Point is struct { X, Y int }
and you build the standard library into libstd.so
and then you build a graphics program against it.
And suppose then the standard library changes to add Z int to Point.
Any stack allocation of a Point in the graphics program is now failing to reserve enough space,
which will probably lead to memory corruption when invoking any function that takes or returns a Point,
or invalid address arithmetic accessing points[i] for var points []draw.Point.
The problem is that the compiled client code assumes something about draw.Point,
namely that it is two words long, so that can't change in libstd.so without recompiling the clients.
Escape analysis, inlining, and other optimizations make the same kinds of assumptions:
if draw.Point.Add has been inlined into the client code, then changing the copy in libstd.so
will have no effect (or, worse, inconsistent effects depending on inlining decisions).
And if a function parameter that didn't escape starts escaping,
optimizations applied in the client code are no longer valid.

These may seem like obvious examples, especially to people familiar with shared library gotchas.
We could of course add even more special logic to the toolchain,
so that it knows where the shared library boundaries are,
and then we could disable any use of escape analysis results
or inlining of function bodies across those boundaries.
But we are never going to disable things like doing a simple multiplication to access points[i].
The result would be too horribly inefficient.
Instead we change the ABI hash when almost any detail of the original source changes,
which means that you can basically never drop in a new libstd.so and use it with
programs compiled against an older libstd.so (#23405).

On top of those fundamental problems, there are many corner cases that don't work,
most of which @seankhliao helpfully closed (thanks!):

We can leave those closed, by the way.

There are two main reasons that people reach for shared libraries.
First, the argument goes, shared libraries make it possible to handle a security fix for a library
by updating the one copy of the library and to avoid updating all programs using it.
Second, shared libraries result in smaller binaries.

The security fix argument basically doesn't hold up at all.
Even in C, it only works if you are incredibly careful in the first place
and don't need to change any details of the library headers, such as
struct layouts, #defined macros, and so on.
In Go, as noted above, it works approximately never. Never has.
Even if we solved the escape analysis and inlining problems,
it's not uncommon to need to add a new field to a struct in order to fix a security problem.
If you have been using Go's -buildmode=shared to make security fixes easier to deploy,
you probably have unpatched or unstable programs and should reconsider that decision immediately.

That leaves smaller binaries.
The last three comments above all mention binary size as the reason for using -buildmode=shared.
And I appreciate that hello world linked against libstd.so is only 1.6% the size of its statically linked cousin!
(A more realistic example: the go command is 31% the size when std is dynamically linked.)

Assuming that smaller binaries is the only use case we need to support,
I think it would work to simplify down to supporting only a single .so,
which you can build with -buildmode=shared but then have to provide
explicitly to future builds as “this is the .so I want to share code with.”
Those future builds read what they need from the .so (bypassing $GOROOT entirely),
and they fail if the .so is stale.

The current idiom

go install -buildmode=shared std
go build -linkshared helloworld.go

would be replaced with:

go build -buildmode=shared -o libmyshared.so std
go build -linkshared=libmyshared.so helloworld.go

Of course, it wouldn't have to be just std: it could be std plus other packages.
But you only get one .so, and you have to manage installing it
and providing it to future go build commands.
All the .shlibname code in cmd/go gets deleted.

Distributions might choose to do

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

And then if they need to support multiple Go versions, it's obvious what to do.

Glancing through the bug titles above, it looks like this design would eliminate
the vast majority of the issues we've had. It definitely addresses the most serious
problems that I mentioned above. The only problem I'd still be concerned
about is the handling of debug information, but that should be doable.

We are already planning to keep -buildmode=plugin and -buildmode=c-shared,
so I am hoping that the amount of extra code for this more limited -buildmode=shared
will not be a significant burden.

To everyone, but especially to @rasky, @zhouguangyuan0718, @Jason7602, and @mlaventure:
would this alternate UX for -buildmode=shared work for your current use cases?

Thanks.

Answer 27 · 2021-10-29T16:43:45.000Z

Bumped this back into "Likely Decline" so that we can include the status update in the next proposal minutes.

Answer 28 · 2021-10-29T17:29:37.000Z

To everyone, but especially to @rasky, @zhouguangyuan0718, @Jason7602, and @mlaventure:
would this alternate UX for -buildmode=shared work for your current use cases?

Thanks for reopening and considering this use case @rsc .

Your proposal would work for me indeed. Not as flexible as having every module as a separate library, but given the added complexity, I find the one library to be a nice compromise. 👍

Answer 29 · 2021-10-29T17:41:18.000Z

Thanks for reopening. It would work for me. My use case is embedded system with limited flash where you want to deploy multiple binaries written in Go. The current workaround is to link all unrelated applications in a single binary and then use symlinks and check argv[0] to internally dispatch to the right entrypoint (ala busybox).

Putting common code (std + other packages) in a single .so file would surely solve my problem.

Answer 30 · 2021-11-02T17:24:54.000Z

Russ wrote:

The only problem I'd still be concerned about is the handling of debug information, but that should be doable

One thing we can do that would make the problem more tractable in general would be to move DWARF type DIE generation out of the linker and into the compiler (this is an idea that we've toyed with the in the past but haven't gotten to). Doing that would definitely help for the "link shared against libstd.so" use case.

Answer 31 · 2021-11-03T18:05:08.000Z

No change in consensus, so declined.
— rsc for the proposal review group

Answer 32 · 2021-11-03T21:04:47.000Z

So... I got excited and intrigued (I want smaller executables too...), and tried

go install -i -buildmode=shared std
go build -linkshared helloworld.go

but getting

$ go version
go version go1.17.2 linux/amd64

$ go run -linkshared helloworld.go
fatal error: unreachable method called. linker bug?

goroutine 1 [running]:
runtime.throw({0x7f6f2569d855, 0x55948fd23ae8})
	/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc00035eec8 sp=0xc00035ee98 pc=0x7f6f24f6f351
runtime.unreachableMethod()
	/usr/local/go/src/runtime/iface.go:561 +0x25 fp=0xc00035eee8 sp=0xc00035eec8 pc=0x7f6f24f39fa5
fmt.Fprint({0x55948fd23b18, 0xc000420008}, {0xc00035ef60, 0x1, 0x1})
	/usr/local/go/src/fmt/print.go:233 +0x75 fp=0xc00035ef38 sp=0xc00035eee8 pc=0x7f6f25083115
fmt.Print(...)
	/usr/local/go/src/fmt/print.go:242
()
	/home/atch/ww/at/go/test/helloworld.go:6 +0x55 fp=0xc00035ef80 sp=0xc00035ef38 pc=0x55948fd211d5
runtime.main()
	/usr/local/go/src/runtime/proc.go:255 +0x282 fp=0xc00035efe0 sp=0xc00035ef80 pc=0x7f6f24f72a02
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00035efe8 sp=0xc00035efe0 pc=0x7f6f24fab4e1
exit status 2

So, question to people who saw it working - when was it? (my understanding is that because this feature is on a borrowed time right now, there is no point to open a bug report)

Answer 33 · 2021-12-04T18:14:33.000Z

Hello,
It's quite funny that Mr. Cox with all due respects, propose removing shared library support, and get it accepted and closed, on the basis nobody uses it,

WHY DO YOU THINK I HAVE BEEN reporting such bugs?
Do you have any legal review board? Obviously not,

What shall we do with legal license requirements that shared library is ok but compiling into static is not?

I think a sane programming language should have shared library support

It seems we have to move to rust as Go is getting crazier everyday

Answer 34 · 2021-12-05T02:02:20.000Z

@kiap I don't know if you realize that this proposal was declined and was not implemented.

There is a new suggestion in #47788 (comment). It would be helpful to know whether that would or would not work for your case.

Answer 35 · 2021-12-06T05:44:11.000Z

I, for one, was unaware of that, despite the fact that I follow the issue tracker fairly closely a lot of the time. I stopped following this one, however, once the accepted label got applied. Could the labels be updated to reflect the actual status?

Answer 36 · 2021-12-06T07:44:54.000Z

To everyone, but especially to @rasky, @zhouguangyuan0718, @Jason7602, and @mlaventure:
would this alternate UX for -buildmode=shared work for your current use cases?

I'm so sorry for my so late reply, @rsc .The new proposal seems more perfect and graceful for smaller binaries.
But I think there is some details to think over and over.

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

If I don't misunderstand, in this proposal, cmdx is a main package, all the dependencies of cmdx should be contained in libgo1.17.2.so. It is a new constraint. Maybe we can support some more packages build in the build command or other way.

And more details I can realize is the dependency chain of dynamiclink.
Will it support a commands looks like go build -buildmode=shared -linkshared=libgo1.17.2.so -o libcmd.so cmd? I think this is useful, And comparing with the new proposal, it will not be a significant burden.

The last thing I'm care about is that if the new proposal is implemented, can it support the plugin with linkshared? Looks like go build -buildmode=plugin -linkshared=libgo1.17.2.so -o plugincmd.so cmd. I had implemented limited "plugin with linkshared" and I'm planing to use it in our producting enviroment. I think if the new proposal is accepted, it will more easy and less situation to resolve for "plugin with linkshared" .

I'm always focus on shared library or dynamiclink in go. And I'am glad to contribute any code about shared library or dynamiclink in the future.

Thanks.

Answer 37 · 2021-12-06T07:46:39.000Z

So, question to people who saw it working - when was it? (my understanding is that because this feature is on a borrowed time right now, there is no point to open a bug report)

@tandr
It is a bug in go 1.17~1.17.2. I had fixed it in go 1.17.3 and master. You can try 1.17.3 or gotip.

Answer 38 · 2021-12-06T13:53:44.000Z

Let me explain: I am quoting from @rsc

There are two main reasons that people reach for shared libraries.
No there are three at least: (or even more)

First: Security fix, one shared library for all of its clients/users code

The argument he gave is it is broken unless you are careful; Sir, you need to stick to SemVer and security fix should not break that compatibility, Go is loosey-goosey, breaking everything of course, (My complaint)

Second: Code size
While this argument is true, I never grasped the idea of one exec file do it all you make, while all bosses and clients and techs used to C style library + executables + other stuff, Deploying a single large exec makes your commercial product weird to them, there are other obvious technical merits, such as space constraint, Which I do not think Go fits in,
When we picked up Go we wanted to see efficiency but got bloated memory usage, did not let us use it for much system-level management of internal Cloud Virtual Machines (VPSes)
The idea seems for Google is memory is cheap so who cares right! To Me Go is thinking of itself as a replacement for Java or Python, we gravitated to it from C, for system-level management and distributed systems management.

Third: Legal issues,
Let's say LGPL license, so many commercial codes or licensed code based on patent and any commercial code that permits only shared linking, don't ask my inlining cross-code optimization and thus break that, tell the lawyers who wrote them, those who want to break free from liability need to avoid derivative work, and thus shared linking is required.

There could be other reasons I cannot think of now, I cannot tell even it is three use cases.

@rsc wrote:

We've decided to leave -buildmode=shared alone (not delete it) for Go 1.18,
but it's not going to work any better than it does today.
In particular, we're not fixing any bugs in it anymore,
and we still intend to remove it in a future release.

This is exactly what I experienced since 2016, every major release, something breaks the shared library and I had to report it to get it fixed. the ultimate one is modules.

@ianlancetaylor no the proposed solution in the comments does not work,

for exactly the same reasons of (1) (2) and (3)

We create a library directory that holds shared libraries in /usr/local/libs/

if that library exists go uses shared otherwise just pie see below: (You already have broken this in every major release and I reported and got it fixed)

even we do not create libstd.so
we create disaggregate libs

This is how we do it: (for every version) below is for 1.17

If you see multiple libraries in ONE LINE OF the list because of Go gotchas, this is the only way you can compile disaggregate (beyond a single comment)

`vi lib_list_1.17.txt

runtime sync
runtime/race
internal/unsafeheader
unicode
unicode/utf8
internal/itoa
math/bits
container/list
container/ring
crypto/internal/subtle
crypto/subtle
unicode/utf16
vendor/golang.org/x/crypto/cryptobyte/asn1
internal/nettrace
vendor/golang.org/x/crypto/internal/subtle
math
encoding
internal/goversion
hash/maphash
image/color
internal/cfg
reflect/internal/example1
reflect/internal/example2
image/color/palette
math/cmplx
runtime/metrics
internal/reflectlite
internal/testlog
internal/sysinfo
internal/singleflight
math/rand
errors
sort
io
internal/oserror
path
crypto/elliptic/internal/fiat
vendor/golang.org/x/net/dns/dnsmessage
strconv
syscall
container/heap
strings
bytes
hash
crypto/internal/randutil
hash/crc32
hash/adler32
crypto/hmac
text/tabwriter
vendor/golang.org/x/crypto/hkdf
hash/crc64
vendor/golang.org/x/text/transform
reflect
crypto
bufio
crypto/rc4
encoding/ascii85
encoding/base32
net/http/internal/ascii
go/build/constraint
regexp/syntax
hash/fnv
html
net/http/internal/testcert
compress/bzip2
image
internal/syscall/unix
internal/syscall/execenv
time time/tzdata
plugin
regexp
image/internal/imageutil
image/draw
image/jpeg
context
io/fs
internal/poll
embed
os
internal/fmtsort
encoding/binary
crypto/ed25519/internal/edwards25519/field
crypto/sha512
crypto/cipher
crypto/md5
crypto/sha1
crypto/sha256
encoding/base64
vendor/golang.org/x/crypto/poly1305
fmt
encoding/pem
crypto/ed25519/internal/edwards25519
net
path/filepath
io/ioutil
internal/lazyregexp
crypto/aes
crypto/des
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/sys/cpu
index/suffixarray
os/exec
internal/obscuretestdata
runtime/debug
os/signal
vendor/golang.org/x/crypto/chacha20poly1305
compress/lzw
encoding/hex
os/user
net/url
compress/flate
math/big
vendor/golang.org/x/crypto/curve25519
database/sql/driver
debug/dwarf
debug/gosym
debug/plan9obj
database/sql
encoding/csv
archive/zip
compress/gzip
compress/zlib
encoding/gob
encoding/json
debug/elf
debug/macho
debug/pe
archive/tar
encoding/xml
crypto/dsa
crypto/elliptic
encoding/asn1
crypto/rand
log
crypto/ed25519
crypto/rsa
vendor/golang.org/x/text/unicode/bidi
vendor/golang.org/x/text/unicode/norm
vendor/golang.org/x/crypto/cryptobyte
crypto/x509/pkix
vendor/golang.org/x/net/http2/hpack
mime
mime/quotedprintable
net/http/internal
flag
vendor/golang.org/x/text/secure/bidirule
go/token
text/template/parse
crypto/ecdsa
internal/buildcfg
internal/execabs internal/execabs internal/goroot go/build go/internal/gccgoimporter go/internal/srcimporter
vendor/golang.org/x/net/idna
go/scanner
go/constant
internal/xcoff
text/scanner
image/gif
go/ast
image/png
internal/profile
runtime/trace
internal/trace
crypto/x509
net/textproto
vendor/golang.org/x/net/http/httpproxy
text/template
testing
vendor/golang.org/x/net/http/httpguts
mime/multipart
log/syslog
runtime/pprof
go/internal/typeparams
net/internal/socktest
net/mail
go/parser
go/printer
os/signal/internal/pty
testing/iotest
internal/testenv
crypto/tls
testing/fstest
testing/quick
go/doc
html/template
internal/lazytemplate
testing/internal/testdeps
vendor/golang.org/x/net/nettest
go/types
go/format
net/http/httptrace
net/smtp
net/http
expvar
net/http/cgi
net/http/cookiejar
net/http/httptest
net/http/httputil
net/http/pprof
net/rpc
net/http/fcgi
net/rpc/jsonrpc
go/internal/gcimporter
go/importer
`

Another bash file (This is an example)

`#! /bin/bash
GO_DIR=/usr/src/dev
LIB_DIR=/usr/local/lib/project
WORK_DIR=/usr/src/project
STD_LIB_LIST=lib_list_1.17.txt

            IFS_saved=$IFS
            IFS=$'\n'
            for i in `cat ${WORK_DIR}/${STD_LIB_LIST}`
            do
                    IFS=$IFS_saved
                    go install -v -ldflags '-s -w' -gcflags "-trimpath ${WORK_DIR}" -buildmode=shared -linkshared -pkgdir "${LIB_DIR}" $i
            done

            IFS=$IFS_saved

`

then to build:

    # -linkshared here  falls back to static linking if the -pkdir is empty or the libraries could not be found
    # we use this auto-detection of shared libraries, to compile shared or static library just by 
    # creating the library directory in Makefile and compiling them using lib option above
    # so if no shared library is found static linking is invoked with pie linkshared  otherwise the available libraries
    # are compiled in as dynamic linking

$2 is your project

go install -v -ldflags '-s -w' -gcflags "-trimpath ${WORK_DIR}" -buildmode=pie -linkshared -pkgdir "${LIB_DIR}" "$2"

All just to satisfy the legal issue!
The build scripts are more complex I just provided a simplified piece

Even our libraries are placed like that

Unless you assume every single piece of code you got is licensed as BSD o MIT or apache or you wrote it then feel free for using static linking.

If you cannot satisfy dynamic shared libraries requirement, Go is at the level of Perl, PHP, in our usage, as a medium-size business we need to satisfy the legal department, I cannot comment on Small Businesses who do not care about legalities.

Answer 39 · 2021-12-06T14:18:00.000Z

Just to comment on my own comment that
The idea of a shared library is to create once and used it as many times

With Go you have to recompile the entire shared library every time you compile (Unless you keep cache) which is more complicated

Go is not C, I see, but cannot change legalities of licenses too.

So it would be very much appreciated if a shared library acted like exactly as C, an OS vendor then shipped one library of a version in disaggregate.

Answer 40 · 2021-12-06T14:28:28.000Z

@DeedleFake Sorry, you're right, the history of this proposal has been confusing.

I removed the "Accepted" label.

Answer 41 · 2021-12-06T14:30:13.000Z

@kiap I would be very interested in hearing of any Go package anywhere that is distributed under the LGPL. Given Go's current technology such a package would effectively be under the GPL, so I don't understand why anybody would choose the LGPL.

If there are no Go packages under the LGPL, then I don't understand the legal concerns.

Thanks.

Answer 42 · 2021-12-06T22:48:28.000Z

@ianlancetaylor No it is not, GPL

Technically even shared libraries in C : part of the code is derived from the original by the compiler,
however:

    "A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being
    compiled or linked with it, is called a "work that uses the Library". Such a work, in isolation, is not a derivative work of 
     the Library, and therefore falls outside the scope of this License. ",  the LGPL version 2.1

the exception for shared library allows the part the compiler or linker mixes in

Sample write a wrapper around libraries in C that are LGPL

There are other works that people write and put under MIT, BSD licenses, if you cannot be sure of the origin then you can be held legally liable regardless of the original author mistake or ill-fated decision,

If there are no Go packages under the LGPL, then I don't understand the legal concerns.

To me, you made your decision Sir, and are simply asking to avoid any other licenses that are not MIT, BSD, Apache, and the like. That's a solution of course why did they clutter the OS with shared libraries, right? perhaps at 46, I am old.

Answer 43 · 2021-12-07T17:46:24.000Z

Writing a wrapper around LGPL C libraries is a "work that uses the library." The C libraries can continue to be shared libraries. There is no need for the Go code itself to be in a shared library. That works today and will continue to work.

To me, you made your decision Sir, and are simply asking to avoid any other licenses that are not MIT, BSD, Apache, and the like. That's a solution of course why did they clutter the OS with shared libraries, right? perhaps at 46, I am old.

That is not our intent. It is fine for Go programs to use GPL libraries. It is fine for Go programs to use LGPL libraries, as shared libraries, if those libraries are written in other languages. It's true that it's hard for Go code to be licensed under the LGPL; it is in effect the same as being licensed uder the GPL. But that doesn't seem so bad as even the FSF recommends against using the LGPL (https://www.gnu.org/licenses/why-not-lgpl.html).

The difficulty with shared libraries in Go is not because we don't like shared libraries. It's because they are technically difficult to support, and nobody has figured out how to do it well.

And for what it's worth I'm a lot more than 46 years of age.

Answer 44 · 2021-12-07T20:32:30.000Z

@ianlancetaylor thanks for addressing our concern but,
A wrapper around the LGPLed C library for easier use in Go, is still a library (If it functions as a library it is a library)

A solution would be directly using that C-shared library with cgo and avoid the wrapper
The difference is the programmers should now know the inner workings of the C library.

the ultimate solution is just to write the entire LGPL/GPL code in Go
Too much work I suppose

The difficulty with shared libraries in Go is not because we don't like shared libraries. It's because they are technically difficult to support, and nobody has figured out how to do it well.

very well understood, Thank you

Answer 45 · 2021-12-08T04:12:31.000Z

A wrapper around the LGPLed C library for easier use in Go, is still a library (If it functions as a library it is a library)

Yes, but the Go wrapper is not a derivative of the C code, and as such the Go wrapper can be under a different license. The C code, under the LGPL, will be in a shared library, and the Go wrapper, not under the LGPL, will not.

Answer 46 · 2021-12-08T18:00:59.000Z

@ianlancetaylor

but the Go wrapper is not a derivative of the C code, and as such the Go wrapper can be under a different license

No sir, that's your technical opinion as an engineer, not a legal opinion

LGPL defines these terms on its own legal universe, Basically, it means you cannot circumvent LGPL by actually writing another library and putting another license to it, doing the same thing, otherwise, this is possible with C, or other languages, if making a library in Go of a library of LGPL, it must be LGPL and be a library, the usage of LGPL directly does not fall under this (as I wrote),

It is well understood legally an LGPL library must be dynamically linked or the application violates this,

This is based on LGPL 2.1 otherwise 3.0 is even more difficult.

A "library" means a collection of software functions and/or data prepared so as to be conveniently linked with application 
programs (which use some of those functions and data) to form executables.

...

 2. You may modify your copy or copies of the Library or any portion of it, thus forming a work based on the Library, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions:

a) The modified work must itself be a software library.

b) You must cause the files modified to carry prominent notices stating that you changed the files and the date of any change.

c) You must cause the whole of the work to be licensed at no charge to all third parties under the terms of this License.

d) If a facility in the modified Library refers to a function or a table of data to be supplied by an application program that uses the facility, other than as an argument passed when the facility is invoked, then you must make a good faith effort to ensure that, in the event an application does not supply such function or table, the facility still operates, and performs whatever part of its purpose remains meaningful.

If Go does not support shared library for Go libraries, it is legally safe to say, not to write wrappers around LGPL C code and statically link it, it is very easy to show the user is violating LGPL

Answer 47 · 2021-12-08T22:39:10.000Z

No sir, that's your technical opinion as an engineer, not a legal opinion

That is correct. However, it is also my informed opinion due to having worked on free software licensing issues for decades. I was one of many people who contributed to the design of the LGPL in the first place.

LGPL defines these terms on its own legal universe, Basically, it means you cannot circumvent LGPL by actually writing another library and putting another license to it, doing the same thing, otherwise, this is possible with C, or other languages, if making a library in Go of a library of LGPL, it must be LGPL and be a library, the usage of LGPL directly does not fall under this (as I wrote),

It is well understood legally an LGPL library must be dynamically linked or the application violates this,

Yes. In the scenario I described, the LGPL code continues to be dynamically linked. Absolutely nothing changes with respect to the LGPL code. No part of the LGPL code is modified in any way. It continues to be under the LGPL, it continues to be distributed as a dynamically linked shared library. It continues to be possible for the end user to substitute in a different version of the shared library when running the executable.

Answer 48 · 2021-12-09T00:44:40.000Z

@ianlancetaylor

Thanks

My problem: Lawyers at our company: Neither GPL nor LGPL has been explicitly tested in a court of law, the linking issue subject to MySQL vs. NuSphere case was settled outside of the court.

So we take the most conservative way; defending in court is too expensive. exceeding the boundaries of compliance to a license is not.

Answer 49 · 2021-12-09T06:18:28.000Z

@ianlancetaylor

Some folks contacted me from ITF USA
Saying the GPL / LGPL problem with Go can easily be fixed in the most conservative way:

Sending messages from process A (EULA or any license) to process B (GPL licensed), embracing 100% separation

I guess that's doable,

Thanks for @ianlancetaylor 's help and other folks for their input.

Answer 50 · 2021-12-17T01:08:49.000Z

@rsc Is there any issue or discussion to track about your -o proposal here?

My so far hypothetical use case is a multiprocess architecture, with a bunch of small helper programs that possibly depend on C libraries and can be installed separately. Again, it's about size.

I get a feeling that Linux distributions could make use of a "go-rt" package for "std" and link their stuff against that. It would save a bit of bandwidth for utilities.

Answer 51 · 2022-01-07T19:40:49.000Z

I've been working on plugin loading and splitting components cleanly (and automatically) across multiple IPC-connected processes, the code is available here: controller-bus.

It could sure use some improvement on the c-shared plugin loading side, (although I know that's unrelated to this issue)

Answer 52 · 2022-01-23T20:04:15.000Z

After reading this whole thread it's not clear to me about -buildmode=shared 's future. Can I summarize it as:

plugin and c-shared will stay.
shared will be limited to use cases as below only:

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

Answer 53 · 2022-01-23T20:39:40.000Z

@laoshaw There has not been a final decision but what you describe is likely. More precisely, it's likely that we will only permit a single -buildmode=shared library to be included in a build; it doesn't necessarily have to be the Go standard library although that will presumably be the common case.

Answer 54 · 2022-06-13T17:20:08.000Z

Just wanted to check on the status of this. Since the issue is closed, is there another issue/location where I can follow the evolution of this? I don't see it explicitly in the 1.19 beta 1 change log so I'm guessing at best it'll be for 1.20?

Answer 55 · 2022-06-13T18:47:10.000Z

@mlaventure I think the status is that we developed an idea (a single shared library) but stalled out on implementing it. Nothing has changed yet and as far as I know there is no new proposal.

Answer 56 · 2022-06-16T14:59:25.000Z

@ianlancetaylor thanks for the update. Is there somewhere I can track the progress on this? In the meantime we'll just have to transition to a different language for our systems with stricter memory constraints unfortunately.

Answer 57 · 2022-06-16T18:20:09.000Z

@mlaventure I don't know of a place to track progress. Sorry.

You should change to a different language if it makes sense for you, of course, but I don't understand the comment. The existing -buildmode=shared continues to work for now.

Answer 58 · 2022-06-16T19:19:20.000Z

@mlaventure I don't know of a place to track progress. Sorry.

You should change to a different language if it makes sense for you, of course, but I don't understand the comment. The existing -buildmode=shared continues to work for now.

Unfortunately it doesn't work for me, the compiler panics when I try to build a package with -buildmode=shared -linkshared:

panic: runtime error: makeslice: cap out of range

goroutine 1 [running]:
cmd/link/internal/loader.(*Loader).LoadSyms(0xc00079a000, 0x670a56)
	/usr/local/go/src/cmd/link/internal/loader/loader.go:2210 +0x9f
cmd/link/internal/ld.(*Link).loadlib(0xc0000da000)
	/usr/local/go/src/cmd/link/internal/ld/lib.go:577 +0x46e
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0x0, 0x0}, {0x679c5b, ...}, ...})
	/usr/local/go/src/cmd/link/internal/ld/main.go:249 +0xd8b
main.main()
	/usr/local/go/src/cmd/link/main.go:69 +0x1005

This is using go 1.17.11.

Answer 59 · 2022-06-16T19:27:45.000Z

You wouldn't normally use both -buildmode=shared and -linkshared together. Anyhow perhaps that should be a different issues if there isn't already one open for it. Thanks.

Answer 60 · 2022-06-16T19:34:28.000Z

That is #47455, which could perhaps be reopened — but, to be honest, I would be surprised if it gets much debugging effort until we figure out the future of -buildmode=shared.

Answer 61 · 2022-06-16T19:40:50.000Z

You wouldn't normally use both -buildmode=shared and -linkshared together. Anyhow perhaps that should be a different issues if there isn't already one open for it. Thanks.

It used to be the only way I had found to build a shared library and avoid the /usr/local/go/pkg/tool/linux_amd64/link: cannot implicitly include runtime/cgo in a shared library error. I think there was an issue for it around go 1.7, but the issue seems to still be there. The packages I am building do not depend on CGO.

Answer 62 · 2022-07-12T18:46:59.000Z

Change https://go.dev/cl/417194 mentions this issue: misc/cgo/testshared: run tests only in GOPATH mode

Answer 63 · 2022-09-26T23:43:46.000Z

I've been looking at -linkshared application building recently to see if it's a viable solution for Linux distribution packaging of Go software. The main workflow concept is this:

Distributions provide a -buildmode=shared package that includes libstd.so (in Fedora this is golang-shared).
```
GOROOT=$(pwd) PATH=$(pwd)/bin:$PATH go install -buildmode=shared -v -x std
```
Applications are built with -linkshared to link against this global libstd.so for the Go standard library.

The idea behind this is to enable updates to Go within the distribution without requiring mass rebuilds of applications due to stdlib patches/fixes. It has the side benefit of creating smaller binaries, but the gains are dependent on other factors as well (e.g. minimal dependencies). However, I do not fully understand the Go build process and the way the compiler works to know if this is even a feasible goal.

As it stands right now this isn't possible because Go attempts to rebuild libstd.so on each build run, which doesn't lend confidence to a stable shared object being produced, and users building software as non-root generally do not have write access to the global GOROOT.
I've been told that it's possible for things to change between patch releases of Go (e.g. 1.19.0->1.19.1) that might introduce reliability issues in this process.

Dependencies using assembly may run into issues

For example, trying to build terraform:

$ GOROOT=/tmp/golang go build -o ./bin/terraform -mod=vendor -linkshared .
# github.com/klauspost/compress/zstd/internal/xxhash
asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access and is used here: 00092 (/home/mroche/Documents/code/tools/terraform/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.s:120)ADDQ	R15, AX
asm: assembly failed

If my understanding is correct, this wouldn't remove the need to rebuild applications if there's a security fix to the compiler itself, only the stdlib.

Does this sound like a use case that could be (or maybe shouldn't be at all) supported by Go tooling?

Answer 64 · 2022-09-28T18:22:25.000Z

@omenos, to my knowledge the Go compiler doesn't currently have any notion of “ABI boundaries”, and in particular it can do inlining of function bodies across packages. So there is a substantial risk of subtle bugs when picking up changes to the standard library and toolchain — particularly if a function fixed in a patch release was inlined into the caller when it was compiled.

Answer 65 · 2023-02-25T08:47:31.000Z

Distributions might choose to do

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

This sounds like a great solution. I know I'm late to the game and perhaps it's a bit much to ask ;) What would really help in my use case (single repo/module many separate applications) is a build command that can build multiple binaries in one go, and intelligently bundles all shared dependencies (or just anything not in their respective main packages) into a shared library.
So basically your solution, with automation so the risk of operator mistake is limited.

Answer 66 · 2023-05-04T07:42:42.000Z

+1 for shared libraries support with module aware versions

I'm using plugins with their dependencies. And they are very fat. I just want to move the dependencies to a shared library. And I don't want to care about a compatibility etc.

Also, root installation is not acceptable for me too. Currently I'm using separate Go compiler (unlike /usr/lib/go).

P.S. only linux + amd64, don't care any other OS or architecture