cmd/go: add global ignore mechanism for Go tooling ecosystem

Question

cmd/go: add global ignore mechanism for Go tooling ecosystem

burdiyan opened this issue 5 years ago · 129 comments

UPDATE: The summary of the accepted proposal at: #42965 (comment).

Problem

For non-trivial (often multi-language) projects it's often desirable to make all the Go tools (including gopls) ignore certain directories.

Some of the examples could be the huge amount of files within node_modules, or bazel-* directories generated by Bazel. This causes many operations with ./... wildcards taking longer than desired. Also gopls often eats up a lot of CPU in VS Code depending on what you are doing.

Prior Art

This is something that has been discussed in several issues before, but seems like people couldn't get agree on a solution.

Some tools started to have their own solutions which causes fragmentation and is cumbersome.

For example goimports have its own machinery for this - .goimportsignore file in this case. But it's not working with Go Modules.

Other tools have a hard-coded list of directories to ignore, like .git and so on.

It seems like having a global solution that all the Go ecosystem could understand would make sense to solve this kind of problem.

Recently a workaround for this was to place a dummy go.mod file in the directories you wanted to ignore. But this is not easily portable between users of the project, because often these directories can be re-created on the user's machine and aren't even checked-in. Asking people to sprinkle some go.mod files all around every time is cumbersome.

@robpike was against of creating more dot files (#30058 (comment)).

Proposed Solution

Here're some of the options that this could be implemented with.

~~Use go.mod file for specifying directories to ignore.~~ (Rejected because go.mod is not a catch-all config file like package.json in NodeJS).
~~Use a separate .goignore file.~~ (This would go against Rob's desire to avoid new dot files, and although being in the spirit with other tools: .dockerignore, .gitignore, .bazelignore, etc. is concerning. The concerns are discussed in this thread).
Use the go.work file that's coming in the next Go 1.18 release.
Have a separate go.ignore file that would specify directories to ignore.

/cc @tj @stamblerre

Answer 1 · 2020-12-03T09:17:46.000Z

But this is not easily portable between users of the project, because often these directories can be re-created on the user's machine and aren't even checked-in. Asking people to sprinkle some go.mod files all around every time is cumbersome.

I'm not sure that I understand this argument. Presumably, it's a program that creates and fills those directories, since they have to contain a significant amount of files for you to really want to ignore them in Go. If they were just a handful of files created manually by a human, it would be a negligible cost for Go to walk those and realise there are no Go packages there.

So, given that it is a program or script creating those large directories, why not add a touch ${dir}/go.mod at the end? That seems easy enough at a high level, at least.

I'm proposing to add this configuration into the existing go.mod file.

This is unlikely to happen, see #42343 (comment).

Another solution could be a global .goignore file. This would go against Rob's desire to avoid new dot files, but would be in the spirit with other tools like that have files like .dockerignore, .gitignore, .bazelignore, etc.

I have to admit that I dislike this option. It's bad enough that all these other tools use separate ignore files.

Answer 2 · 2020-12-03T13:25:07.000Z

I'm not sure that I understand this argument. Presumably, it's a program that creates and fills those directories, since they have to contain a significant amount of files for you to really want to ignore them in Go. If they were just a handful of files created manually by a human, it would be a negligible cost for Go to walk those and realise there are no Go packages there.

So, given that it is a program or script creating those large directories, why not add a touch ${dir}/go.mod at the end? That seems easy enough at a high level, at least.

@mvdan It is indeed a program that creates these directories. But it's a program that you don't control normally. Wrapping well-known tools like nom install with your own script only to put an empty go.mod in there doesn't seem right.

On the other hand by placing arbitrary files in these directories you're invading the territory of other tools. What if that program checks the integrity of the directory and would break seeing a random unknown file? It's not the case with node_modules but breaking into structures created by other programs, only to work around your own problem doesn't seem right either.

I understand the objection about go.mod. I was not aware about @rsc's statement.

I have to admit that I dislike this option. It's bad enough that all these other tools use separate ignore files.

Could you elaborate on why do you think it's bad? It may not be the most elegant solution, but it's common practice, well-understood and somewhat expected.

If we already have .goimportsignore, why not standardizing it into something that can be handled and understood by all the ecosystem of Go tools?

Answer 3 · 2020-12-03T15:59:35.000Z

It is indeed a program that creates these directories. But it's a program that you don't control normally. Wrapping well-known tools like nom install with your own script only to put an empty go.mod in there doesn't seem right.

Wouldn't you need to wrap the tool to add a .goignore file anyway? (Given that you need to inject a file, why does it matter whether it is named .goignore or go.mod?)

Answer 4 · 2020-12-03T16:13:53.000Z

@bcmills My proposal is to add a file in the root of the project, not in the directory being ignored. So it would be checked in. Like .gitignore in Git. Basically the idea is to list the paths to ignore in that file, and check it in.

Answer 5 · 2020-12-03T16:15:54.000Z

...ok? But why would you not also check in the injected go.mod files?

Answer 6 · 2020-12-03T16:22:06.000Z

Could you elaborate on why do you think it's bad? It may not be the most elegant solution, but it's common practice, well-understood and somewhat expected.

The common practice is to litter repositories with dot files. That does not mean we should do the same, making the problem worse :) Go already has multiple mechanisms to ignore entire directories (. or _ prefixes, and dropping empty go.mod files), so there needs to be a really good reason to add another method.

Answer 7 · 2020-12-03T16:26:03.000Z

...ok? But why would you not also check in the injected go.mod files?

@bcmills because often directories to ignore aren't checked in.

Answer 8 · 2020-12-04T17:28:08.000Z

The common practice is to litter repositories with dot files. That does not mean we should do the same, making the problem worse :) Go already has multiple mechanisms to ignore entire directories (. or _ prefixes, and dropping empty go.mod files), so there needs to be a really good reason to add another method.

@mvdan IMHO, having a dot file in one place, that is trackable, is less of an evil, than sprinkling empty go.mod files all over the place, ad-hoc, and breaking into opinions of other tools.

Thinking about pros and cons of implementing such a feature, I'm struggling to see any cons (probably due to my ignorance), besides having to spend the time to implement it. I'd appreciate if anyone could bring some light into this to understand the implications.

Answer 9 · 2021-03-19T03:38:25.000Z

I have a use case where I have a multi-language repo where not all of the developers are touching the go components.

I don't think it is reasonable to ask my docker, bazel, and nodejs developers to all wrap their normal tooling in scripts that touch extra files in their build directories, nor ask them to try to rename their standard build directories to match existing go conventions, some of which conflict with the other tool conventions.

It seems like there should be a way to specify how to ignore certain files or directories that does not require modifying the content of those files or directories, because the ignored content is not being managed by go and may have its own conflicting conventions and lifecycle.

Answer 10 · 2021-08-18T18:09:18.000Z

@psigen Go wants a directory tree that belongs to it. In a multi-language repo, why not create a top-level go/ directory?

Answer 11 · 2021-08-21T01:08:17.000Z

@rsc: because my projects are not organized that way. I have services like:

service1/
    backend/  # golang
    debug-cli/
    proto/
service2/
    backend/  #python
    debug-ui/
    proto/
webapp/
    frontend/

I know what you are asking, which is why not reorganize to:

proto/
    service1/
    service2/
golang/
    service1-svc/
    service1-cli/
python/
    service2/
nodejs/
    webapp/
    service1/debug-ui/

And the answer, (besides "that's a lot of work right now") is that it is not how our ownership is structured.

It is not convenient to have duplication in the CODEOWNERS files, .gitignore patterns that look like **/service1/foo*, cross-directory-tree docs links, etc. all in service of golang. It makes PR reviews harder when related changes happen all over the directory tree. It forces docker build contexts to all need to be at the root of the entire source tree, and makes live-rebuilds in tools like Tilt and Skaffold much more difficult to author.

I could go on, but I'm really just reiterating the core premise of this proposal:

For non-trivial (often multi-language) projects it's often desirable to make all the Go tools (including gopls) ignore certain directories.

Answer 12 · 2021-09-03T17:57:43.000Z

I use the serverless framework to deploy lambdas on AWS. Some plugins that I have to use contains go files. So when I run go mod download or go mod tidy I add dependencies to my go.mod file that are required by the go files inside the node_modules directory. It would be great to define a way to exclude directories from go modules.

node_modules
  serverless
    lib
      plugins
        create
          templates
            aws-go
              main.go
cmd
  main.go
pkg
  dirA
  dirB
  dirC

Answer 13 · 2021-09-03T23:04:34.000Z

@rsc

@psigen Go wants a directory tree that belongs to it. In a multi-language repo, why not create a top-level go/ directory?

Reading this statement does not make me happy. It goes against the entire premise of the code organization at my company.

We use a monorepo with projects in several languages. Some Go programs live inside projects written in other languages. Sometimes we rewrite a project from one language to another. Some projects use a combination of Go and other languages (imagine a website written using both Go and JavaScript extensively). We already have an organizational hierarchy within the monorepo that is based around purpose and ownership, not language.

This all worked fine with $GOPATH: the repo was inside its own single $GOPATH segment and a top-level vendor directory contained a single version of all shared dependencies.

Moving to modules has raised some challenges, but mostly it has worked. The whole repo is one module so we use a fixed, shared set of dependencies. One issue we faced is that the go mod tidy and other commands printed out a bunch of irrelevant spam (#35941) -- we sent a fix for that. Other issues we see involve various kinds of slowness in gopls (#46438 describes one particular issue I have). But mostly it works fine, and ISTM that the remaining issues are surmountable if the folks working on the tools care about making them work well in the presence of mixed-language source trees (and until now it seemed to me that they mostly do!).

But when I read "Go wants a directory tree that belongs to it", it sounds like you don't think this use case matters as far as the standard Go tools are concerned. I don't know how we could possibly adapt our repo to a "Go code all belongs in its own tree" model. Probably we wouldn't -- I imagine that if push came to shove, we'd look into alternative build tools.

Answer 14 · 2021-12-16T18:08:56.000Z

In #50225 I'm bring in concerns about the resources (network, disk space) wasted on every developers machine because the module zips contain many irrelevant files.

Check this list of files that are in your Go modules cache:

find $(go env GOMODCACHE)/*.* -type f ! -name '*.go' ! -name 'go.mod' ! -name 'go.sum' ! -name 'list.lock' ! -name 'v*.mod' ! -name 'v*.info' ! -name 'v*.zip' ! -name 'v*.ziphash' ! -name 'v*.lock' ! -name 'LICENSE*' ! -name 'README*' -print

I have more than 200,000 useless files on my machine.

This also impacts CI builds (download time/space of new dependencies, requires to enable strong module caching to reduce the problem).

Answer 15 · 2021-12-17T09:46:24.000Z

While similar, I think this proposal is a bit different from yours @dolmen in a sense that here I mostly care about ignoring directories, not specific files, and definitely not for specific packages like x/mod/zip. Still, it could be the same solution for solving both problems.

Answer 16 · 2021-12-17T09:48:36.000Z

BTW, go.work is coming in the next Go release. Maybe this feature could be implemented in there eventually? Or maybe a separate go.ignore file? Looks like a better approach than .goignore for sure!

I updated the initial comment.

Answer 17 · 2021-12-17T11:22:20.000Z

BTW, go.work is coming in the next Go release. Maybe this feature could be implemented in there eventually?

I consider go.work as a development tool for your local development environment. Which means it is a file I would not commit in the repo.

Instead, ignore patterns must be available for tools that download the code from a VCS (for publishing on a proxy, or for filling the module cache, see #50225), so the ignore patterns must be always available in the repository.

So go.work would not be a good place for ignore patterns.

Answer 18 · 2021-12-18T23:09:35.000Z

@dolmen While I suspect that go.work is meant to be checked-in (I'm not sure about it), I think you're right that the ignore stuff should probably be in a separate place, because not all project would want to have go.work. Then maybe go.ignore is the remaining option that would make some people happy, and the rest (those who don't like the idea of dot files) at least not angry about it :)

Answer 19 · 2021-12-19T13:51:41.000Z

@burdiyan, go.work is indeed not meant to be checked in:

These go.work files should not be checked into the repositories so that they don‘t override the workspaces users explicitly define. Checking in go.work files could also lead to CI/CD systems not testing the actual set of version requirements on a module and that version requirements among the repository’s modules are properly incremented to use changes in the modules. And of course, if a repository contains only a single module, or unrelated modules, there's not much utility to adding a go.work file because each user may have a different directory structure on their computer outside of that repository.
— Proposal: Multi-Module Workspaces in cmd/go §Multiple modules in the same repository that depend on each other

Answer 20 · 2022-02-04T11:07:03.000Z

It is kind of frustrating that the responses from Go contributors are uniformly "Everyone else on earth is wrong, they should change to accommodate our design choices."

No matter how inelegant another dotfile is, it solves the problem in a universal way that will work for all repository structures and build tools. None of the proposed alternatives even attempt to do the same.

I currently just don't run gopls and try to minimize how often I have to write Go, which is not a "solution" that is available to everyone.

Answer 21 · 2022-02-18T14:53:07.000Z

Some of us discussed the problem this proposal aims to address - i.e., allow to exclude certain directories when running go with patterns including ....

We agree this is a problem for some tools (e.g. gopls, and others that accept go's import path patterns). Many tools developed their own ways of configure exclusion rules (e.g. gopls has directoryFilter) but this is still not sufficient if they depend on go invocation with ... pattern underneath.

@bcmills had a great idea during the discussion - go already has the overlay mechanism (see the summary of the feature by @matloob and also the -overlay flag description in go command help page). That can be used as the directory exclusion mechanism. For exclusion, place an empty value; for inclusion, set identity mapping. gopls can implement this by applying already existing directoryFilter, and I guess other tools can do the same. (x/tools/go/packages supports overlay)

The overlay config isn't as flexible as glob patterns many dotfiles accept, but I think it still provides the sufficient knob
tools can play with. What do you think?

#50225 (for mechanism to fine tune the scope of a module) was mentioned during the discussion, but I don't think that is the goal of this proposal. For example, I think it's possible one wants to speed up gopls by excluding a directory but want to still keep it in the distributed module (directories containing asset files, etc) or the directory doesn't affect module distribution at all (ephemeral directories such as node_modules or bazel directories created during build).

@jaronsummers I think the Go team is trying to understand the problem better, not dismiss or ignore problems users are facing in the real world.

Answer 22 · 2022-06-16T02:50:00.000Z

I'm sure this was already mentioned here, but for clarity:

The most simple example case of this is if you have node_modules which happens to have any Go code in it. When running "go mod tidy" the Go files in node_modules are scanned & included in go.mod. ignoring node_modules would be the most obvious application of some .goignore feature.

Answer 23 · 2022-06-16T10:29:22.000Z

it was mentioned in the past but adding ignore support to go.mod would be flexible, no hardcoded rules, and no new magic files.

Answer 24 · 2022-06-16T10:57:38.000Z

@amery

it was mentioned in the past

And it was already rejected in the past:

... because go.mod is not a catch-all config file like package.json in NodeJS
— #42965 (comment)

Answer 25 · 2022-06-16T11:18:08.000Z

@antichris every solution has been rejected because developers don't recognize the problem. go.mod is not a catch-all and .goignore is.. another file

Answer 26 · 2022-06-20T08:28:14.000Z

I'd vote for go.ignore! It's another file, but it's not a dot-file, which was the main concern of Rob, and others I believe.

Answer 27 · 2022-06-20T08:51:13.000Z

I'd vote for go.ignore! It's another file, but it's not a dot-file, which was the main concern of Rob, and others I believe.

as long as it can be used to specify patterns to ignore I'm happy

Answer 28 · 2022-06-20T09:15:10.000Z

How about a go env variable, e.g. GOIGNOREFILE, that could be set to point to an arbitrary ignore pattern file with .gitignore-compatible syntax?

Users could then assign GOIGNOREFILE=go.ignore (or .goignore or even .gitignore), if they chose to do so, and no new "another" (dot or not) file is forced on anyone out of the box. It would also be possible to settle on an OOtB default value for it eventually, without breaking established workflows, yet providing a much needed relief in the meantime.

Answer 29 · 2022-06-23T03:10:27.000Z

Surprised there wasn't any devops use cases yet, so I'll step in.

Many great projects are based on Go, among which are - docker, kubernetes, helm and terraform. It is only so natural that many of us dealing with these tools on a daily basis becoming fluent with Go over time, and starting to use it more. Particular example of that is Terratest - a great framework that is used to write integration tests for Dockerfile, helm charts and terraform modules. Terratest itself is a Go module so the tests are executed with go test ./....

What that means is that every helm chart or terraform module repository is initialized as a Go module, despite that it doesn't have any Go code (other than tests).

I saw somewhere a mention that Go excludes folders named with . in the front. I am not sure how but it doesn't really happening. Maybe it is just gopls specific issue - I don't know. Terraform creates .terraform folder under which it creates sub folders and checking out other modules this module depends on. My VScode in gopls mode consumes a lot of CPU and generates a lots of warning for duplicated modules because TF modules in my VScode workspace indeed are duplicated under .terraform folders of other modules that are using them. This makes the whole setup so painful. And what if some other tool like terraform were to use a folder without . in the front, just like npm, and there wasn't a way to modify its behavior?

This is such an easy feature to add, that this discussion spread among multiple tickets already consumed x10 times of everyone's time than would otherwise take one single person to just implement it. I hope my struggle is not for Go developers arrogance and they indeed trying to understand the problem, but somehow I find it hard to believe. I feel like to address this issue we first will need a help of a licensed therapist that would conduct a series of sessions with the maintainers and help them understand that despite their awesomeness and undeniable historical contribution to the humanity legacy - the universe does not spin around them and there are other tools in existence that Go needs to peacefully coexist with.

Answer 30 · 2022-06-23T03:16:20.000Z

@dee-kryvenko Please be charitable and respectful, per the Go Community Code of Conduct. Criticize the arguments, not the people. Thanks.

Answer 31 · 2022-06-23T03:31:02.000Z

I understand that sometimes, shielded by code of conduct from all criticism and insecurities, by depriving yourself from reality and feedback, it is easy to get disconnected from it.

So far all arguments been denied and no counter arguments or proposals been put forward. In the absence of arguments and ideas to criticize I will criticize the people blocking all other ideas, and I am not alone. What choice do you leave us? What else can we possibly criticize after 2 years in this ticket and several more in others? The problem swept under the rug will not go away by itself. Sooner or later the time will come to confront it. Eventually you will have to reconnect with reality or ban many of your users, possibly contributors, don't you think?

This is such an easy thing to implement that a 6yo could do it. But nobody will want to waste their time on a PR that is clearly not welcome and will not get merged, not accordingly to a several years worth of discussions.

I rarely read CoC but just for fun I just went and read it. I fail to see arrogance and ignorance among the protected characteristics (yet) so it appears to be that I should be fine. As long as the judges or the committee or whatever they call themselves are based and hold on to their own standards. I obviously did not insult anyone and I find my messages a healthy criticism and a wake up call. What could I possibly gain from a random conflict on a random messaging system with some random strangers? My only intention is to get this issue fixed, because my own quality of life at work depend on it. I am sick and tired of how much gopls drains my battery and how hot my laptop gets when I hold it on my laps every time I edit terraform code in my VScode. I find anyone not seeing it that way in violation of a CoC, specifically the part where maintainers put a responsibility onto themselves to "Gracefully accepting constructive criticism". Because yeah - CoC is not only rights and protections but also responsibilities.

Not to mention that I didn't forget to pay my respect, note how awesome Go is, its authors are, and how big and undeniable their contribution to the future of humanity. This and the criticism is not mutually exclusive.

Answer 32 · 2022-06-23T10:10:23.000Z

@dee-kryvenko your last comment is off-topic for the proposal at hand, so I've hidden it to keep the thread on topic. If you have any thoughts on the code of conduct, I would suggest to email conduct@golang.org or golang-dev@googlegroups.com.

I saw somewhere a mention that Go excludes folders named with . in the front. I am not sure how but it doesn't really happening. Maybe it is just gopls specific issue - I don't know.

If that is happening, that is a separate bug that needs to be fixed. Please file it as a new issue with details.

Answer 33 · 2022-06-24T00:23:42.000Z

A lot of gopls' problems mentioned here can be fixed by improved handling of existing mechanisms: go.work files and directoryFilters settings. We're aware of several places where those settings are not handled correctly, resulting in gopls loading unnecessary data. Several of these will be fixed in the next gopls release (v0.9.0). Others will take longer.

In particular: one missing piece is hiding directories filtered by directoryFilters from the go command using overlays, as suggested by @bcmills and described by @hyangah in #42965 (comment).

Given that go.work and directoryFilters already exist, I think we should mostly disconnect this proposal from gopls' performance problems: having a global ignore mechanism does not necessarily solve gopls' problems, nor are gopls' problems unsolvable without a global ignore mechanism*.

*I'll caveat that it may still be that the integration of go.work+directoryFilters is not great, for example because of our inability to translate wildcard filters to overlays.

Answer 34 · 2022-07-21T09:44:33.000Z

I see intermittent test failures when files in node_modules change while thing like go test run (here in a parallel CI job):

go test -timeout 360s -cover ./...
pattern ./...: open node_modules/esbuild-darwin-64: no such file or directory

There absolutely must be a way to get go tooling to ignore certain directories and files. node_modules can easily contain upwards of 100k files, the presence of it will slow down all go commands that search for go files by at least a few seconds even on a fast SSD.

Answer 35 · 2022-07-21T11:40:40.000Z

@silverwind

node_modules can easily contain upwards of 100k files

... and hundreds of megabytes, even gigabytes per project instance. This bullcrap has always been one of the reasons why developing anything with Node.js chortles balls and why I've been doing all in my power to stay away from all that muck. Is npm's the most retarded dependency management scheme? Maybe not, but nowadays it seems like "everyone else" is doing something more sane. It's a pity that Node.js centric conventions such as CommonJS have poisoned the JS and overall frontend ecosystems, so that one cannot simply switch to Deno.

Answer 36 · 2022-07-21T12:25:04.000Z

Again, please keep it civil.

And please avoid adding comments that repeat what has already been said. To add your +1, add a reaction at the top: https://github.com/golang/go/wiki/NoPlusOne

Answer 37 · 2022-12-03T19:34:52.000Z

Here's a simple workaround for go generate .... It should be as ~performant as Go natively supporting .gitignore.

$ go generate $(git ls-files -c -o --exclude-standard | grep "\.go$" | xargs dirname | xargs -I {} echo $(grep module go.mod | cut -d ' ' -f 2)/{})

Answer 38 · 2022-12-05T22:47:54.000Z

This might not solve all the problems here but if you are working with npm together with go, it is better to separate them
in their individual directories and still track them by git.
e.g.

/<project-directory>
  |_ .git/
  |_ go-src/
    |_ go.mod
  |_ js-src/
    |_ package.json

I used this pattern with CDK and it is working great for me.

Answer 39 · 2022-12-06T07:09:37.000Z

This might not solve all the problems here but if you are working with npm together with go, it is better to separate them in their individual directories

This has a multitude of drawbacks, one being unable to go:embed files in the js directory because go:embed can not embed files from a parent directory (that's a separate issue, thought), or complicating make-based operations because you now need to cd around. I'd classify it as a workaround with too many drawbacks.

Answer 40 · 2022-12-06T22:52:22.000Z

This has a multitude of drawbacks, one being unable to go:embed files in the js directory because go:embed can not embed files from a parent directory (that's a separate issue, thought), or complicating make-based operations because you now need to cd around. I'd classify it as a workaround with too many drawbacks.

I think all these complexities arise because we ignore separation of concerns in our codebase.
If we were to use npm run in Javascript codebase and make in Go codebase, we won't need to cd around in make targets.

Also to solve the issue with go:embed, we could simply configure a Javascript builder to store the final build inside the Go project.

Have been said that, a Go native ignore mechanism would definitely make life easier.
Till then I will be using above solution for my multi language projects.

Answer 41 · 2022-12-07T03:04:12.000Z

configure a Javascript builder to store the final build inside the Go project

Which would force that to stay aware of the location of the Go project, thus breaking that separation of concerns again — back to square one.

The good news is, it's starting to look like Deno could be coming to the rescue for TS/JS content after all, since as of 1.28 it has an already somewhat decent NPM module support that doesn't crap node_modules all over your project directories. (I'm still poking around to see how well it works for my use cases, so don't ask me if it is going to work great for your specific project — you're gonna have to try it on your own, if you really wanna know. Also, I'm so not a Node.js/NPM/frontend-stack person.)

A day may come when denoland manages to whip it into a shape where it is possible for all projects to flush the Node.js-based NPM down to where it belongs — now, that would be a day of great jubilation and festivity. 🥳

Answer 42 · 2022-12-09T21:21:20.000Z

I'm on team go.ignore. I don't understand the arguments against it, and from what I can tell they boil down to three main points:

Adding another file is less than ideal, especially a dotfile
You can add blank go.mod files, and write wrappers around everything to re-make them
Node/Bazel/Whatever should change their standards to make them compatible with existing go behavior (like . and _ prefixes)

To me, these seem like fundamentally contradictory points. How is adding one traceable, consistent, commit-able, and obvious go.ignore file to the root worse than adding and re-adding a bunch of go.mods all over the place (which is also a misnomer, as they're not go modules)?

A go.ignore file doesn't need to be in every project, only the ones that need it. The current solution is for projects that need it to write wrappers around other non-go commands, which is, for obvious reasons, not viable.

Reading issues on this topic has been incredibly frustrating, because everything gets shut down by those three points, but those points are contradictory and non-viable from my perspective.

Currently my solution is a clunky python wrapper around go that implements go.ignore and it is incredibly nice to use as someone who often makes go backend, web frontend projects. I can't ever commit these files, so they are currently only in my personal projects that I never plan to release.

We need a better solution, that much is clear. At this point I don't care if its go.ignore, .goignore, go.mod, go.work, etc. anything is better than the current situation.

Answer 43 · 2022-12-09T23:34:16.000Z

I agree with @calvinlarimore I'd rather add a .ignore or .goignore or go.ignore file than add empty go.mod files

Answer 44 · 2022-12-10T00:25:44.000Z

I agree with @calvinlarimore I'd rather add a .ignore or .goignore or go.ignore file than add empty go.mod files

one just can't assume control over the directories we need go to ignore as go.mod advocates believe

Answer 45 · 2022-12-10T23:50:23.000Z

To me, these seem like fundamentally contradictory points.

It is to you, but not to them. You are missing the point that they think within the bubble of their language and their ecosystem, and they couldn't care less about everything and everyone else. They got to keep their code ideologically clean - all these other tools surely can cater to their purist desire and implement dirty workarounds on their side, can't they? This is how self-righteous ignorant self-important authoritarian communities with a lack of self-awareness operate, otherwise it would be impossible not to see a common sense everyone else here can clearly see.

Answer 46 · 2022-12-12T23:46:03.000Z

@dee-kryvenko Please follow the Go Community Code of Conduct. Please be charitable, respectful, and constructive. Thanks.

Answer 47 · 2023-03-14T11:46:31.000Z

I don't understand why our Go community isn't able to add such a common feature.
There are .gitignore, .npmignore, .dockerignore, etc. Why couldn't there be a .goignore? Why would we want to use some terrible workarounds such as empty go.mod when everyone else is already using the .ignore mechanism with success? I think we should add both .goignore and .gokeep to let us all be able to customize our builds as we wish.

Answer 48 · 2023-03-14T11:51:59.000Z

and if they don't want more files we could have a section in our go.mod indicating directories to ignore. this is useful when storing releases in the proxy too

Answer 49 · 2023-03-14T17:17:14.000Z

These options have been discussed previously, and dismissed for multiple reasons. It seems like the least "evil" option was to have a go.ignore file, similar to other go.* files we already have.

Answer 50 · 2023-03-14T17:19:06.000Z

These options have been discussed previously, and dismissed for multiple reasons. It seems like the least "evil" option was to have a go.ignore file, similar to other go.* files we already have.

at this point we would take anything other than dropping go.mod files on the directories we need to ignore

Answer 51 · 2023-04-10T08:42:50.000Z

Got hit by this again. We have a multi-language repository, and we declare dependencies for all the languages at the root of the repository. It's not very practical to have a subdirectory for each language, because sometimes things get mixed and are grouped by "domain boundaries" rather than programming languages.

So we have JS node_modules directory at the repo root as well, which contains a @vercel/go NPM package (which seems like an example Go app to deploy in Vercel, I don't know why would Vercel package this as an NPM module). So this thing has Go files in it, and whenever we run go mod tidy at the repo root, the toolchain traverses node_modules and gets confused with those go files inside.

This is really annoying. Most other programming languages have some way to configure include/exclude patterns for paths (paths, not anything like we have for including/excluding modules).

What is missing for at least selecting this proposal to the "Active" review tab, so it at least gets reviewed and discussed on a regular basis?

Answer 52 · 2023-04-10T19:40:20.000Z

this topic made me realise how similar Go and Lua are. if something doesn't fit how the owners work, it's just ignored.

Answer 53 · 2023-04-10T21:50:10.000Z

this topic made me realise how similar Go and Lua are. if something doesn't fit how the owners work, it's just ignored.

Careful, the ~~dictators~~ owners are not fans of free thinking. You just committed a thought crime, and the Ministry of Love representative @ianlancetaylor is about to knock on your door in the night, and the next morning you will be nowhere to be found. Or did you ever existed? Well, that'd be a job for the Ministry of Truth to correct...

Answer 54 · 2023-04-10T22:13:06.000Z

Amusing, but please do follow the Go Community Code of Conduct on this issue tracker. Thanks.

Answer 55 · 2023-04-10T22:23:00.000Z

@dee-kryvenko I see that you have been warned before on this issue, including by me. Attacking individuals is not going to lead to a solution here. Quite the reverse. Please be courteous.

Answer 56 · 2023-04-10T22:47:02.000Z

@dee-kryvenko I see that you have been warned before on this issue, including by me. Attacking individuals is not going to lead to a solution here. Quite the reverse. Please be courteous.

I don't know what's more ironic at this point. The fact that the reverse of rejecting every proposed solution borderline to even recognizing the very existence of the problem - would be to actually do ANYTHING productive, or the fact that the owners are unable to comprehend how at this point they became a problem here. That the only thing left to the entire community now is to look for ways to work around the owners and try seed them an idea that they would think was theirs to begin with so it wouldn't trigger their ego. And in doing that they (the community) somehow violating the code of conduct. Does the code of conduct already says it in plain text that the owners are always right, or is it still obscured?

Answer 57 · 2023-04-10T22:54:24.000Z

@dee-kryvenko It is possible to disagree pointedly and directly but politely. You are not doing that.

Answer 58 · 2023-04-10T23:10:21.000Z

@dee-kryvenko It is possible to disagree pointedly and directly but politely. You are not doing that.

Disagree with what? You are not proposing anything. This would have been implemented by the community ages ago if someone would indicate the willingness to merge the work. You are missing the point that this - what happens here - this is no longer "open source". Community members tried to object without pointing out the obvious - absurdity of the situation created by ego and arrogance of the "owners". Disabling my filters usually is not a first resort for me, but try to think for a split seconds - what other choice does the community has at this point, after years and years of polite endless discussions that led nowhere?

Answer 59 · 2023-04-10T23:22:00.000Z

You just think about this. #16417 is 7 years old. In these seven years - I moved from Ukraine to New York, from New York to California. I changed 3 jobs. I started and finished dozens of projects. If I had a kid that year - my kid would be going to school already. Is this really such a hard feature to implement? Harder than it was to create Go to begin with? Harder than it was to create Kubernetes, Google, Facebook, Microsoft? Because none of these things took 7 years. Do you really not see the absurdity?

Answer 60 · 2023-04-11T11:47:10.000Z

@amery @dee-kryvenko This kind of contributions are the ones every maintainer looks forward to waste time on.
This is open source, go ahead and take the code. You are free to implement it for yourself if you want it, sounds like you are that skilled and can implement it within your coffee break. Feel free to share it with others by contribute it back to the community, or simply use it for yourself and be happy.

Open source maintainers are only humans who spend their free time on open source projects like this, next to their day job and activities, and are therefore only available to a limited extent to work on projects. You are not entitled to free support and open source project maintainers owe you nothing!

As a maintainer of (larger) projects this kind of noise is the reason why things get stuck and the motivation shrinks on some days. Please, for the sake of being (more or less) a grown up human, who can discuss in a logical way, keep on the topic and contribute content and/or information that help to solve this.

Sorry for the off-topic to everyone who subscribed to this topic and gets tired of being notified for nothing.

Answer 61 · 2023-04-11T14:39:24.000Z

@svengreb a lot to unpack here, but let's just say that your generalized outlook on the Open Source is very shallow and one sided.

Let me start with this - not all Open Source is equal.

As a maintainer (of a small projects) myself - let me tell you, our ego is enormous. We are the centers of our own universes, and we are constantly busy, burnout, lacking motivation, harassed by users, unfairly treated by users, under appreciated (by users). Often misunderstood. Nobody in the world understands our struggle, and our huge contribution to the society. Especially these greedy corporations exploiting our work to make millions while we will never see a dime from it. And we are always right. Because - what the hell does these users know, after all?

As a regular contributor (to large projects) - let me tell you a different story. Few years ago - I reported an issue to a Terraform AWS Provider tracker. It was unable to rename an existing RDS without destroying and re-creating it. I then spent a few hours of my evening and implemented a fix, and offered them a PR. A PR that years later remains unmerged, so as the Terraform itself - unable to perform a simple operation of renaming existing RDS. I intentionally am using a very trivial issue that requires only trivial code change to fix it, so nobody can argue that the reason it was never merged was because my code sucks or that my ask was controversial to begin with. No - it was simply nowhere near the list of priorities for the owners. You like to speak of motivation - how motivated do you think I feel to ever contribute to Terraform ever again? Or any other project to that matter, before I receive any sort of commitments or assurances that my effort is not going to be in vain?

Now, let's talk about the perspective of a regular user. It is nice you brought up Homebrew - I like that project a lot. It is Open Source, and it is great. How about Terraform? Is it Open Source? Hell no! Terraform is a commercial product, owned by a corporation, and a corporation decides what goes in and what doesn't. And the criteria for these decisions is very simple - what is good for business? (Trust me - as a creator of an alternative Open Source state file "as a service" backend that is free of dependencies and is "bad for business" for Hashicorp Cloud - I know a thing or two about that.) That corporation chose to publish their commercial product under Open Source license, and they are benefiting from doing that. They get free advertisement, free code reviews and audits, free contributions even. OSS adopters, after all, are great leads to upsell commercial product later. I will not deny that OSS Terraform users are not coming out of that deal empty handed - they are benefitting a lot, and overall - it is a healthy win-win partnership. But is Terraform truly Open Source?

See, there are a number of large but truly OSS projects out there. Kubernetes would be one example (even though that just like Terraform - it was originally developed and then open sourced by a corporation). Just about any other CNCF member would make a great example of a truly Open Source project. What makes a project truly Open Source? I, as a user, can join a SIG, put in effort, make contributions, get recognized and invited into a board. I can then influence how the project evolves. Can I do either of that with Terraform? Even if I were to join Hashicorp? Well, you may want to ask Pulumi founders about that.

So, how about Go? Is it more like Kubernetes or more like Terraform? I honestly do not know much about the structure of Go community, but from a perspective of a user subscribed to this issue for 7 years - it surely feels more like Terraform.

Now, let's talk about entitlements and responsibilities. With a great power - comes great responsibility. Unfortunately, we live in the culture where increasingly people keep talking about rights and entitlements, but no one wants to talk about responsibilities. No one wants to take them. No one even accepts anymore that responsibilities are inseparable from rights and entitlements.

This is open source, you say, go ahead and take the code, you say. Can I, really?

Can I do that with Terraform, for example? That's what Pulumi founders did, after all, after realizing that they have unreconcilable differences with Hashicorp at large. Good for them, I'd say. I am truly cheering for them. Stick it to the big guy. But I genuinely keep wondering - will Pulumi ever survive the lawsuits that are surely coming if they ever to grow into a real competitor to Terraform? Or will it even get absorbed back into Terraform one day, when Hashicorp finally overcome their ego and recognize that not every intent can be described with HCL? But let's not speak of them - let's speak of me. Can I do it? Can I do it in a context of a smaller issue, like that simple RDS renaming? Well, not really. See, unlike some hot headed minds around here - I am conscious not only of my rights to fork something, but also - of my responsibilities that comes out of doing that. Do I really have resources and free time to maintain an entire Terraform AWS provider, not just this one feature?

Yeah. Grim realization. Monopolies and anti-competitive practices exists in the Open Source too! Who might have thought? It is so convenient to deny their existence, since all the code is open, right there in the public. But devil is in the details.

So let's talk about Go. Let's say I were to finally come to terms with unreconcilable differences I have with the owners, and either part ways and switch to some other language - or dare I say, fork Go. Let's, for the sake of the argument, pretend that I even have resources to do the latter. Can I really do that, without simultaneously parting ways or forking everything else I regularly contribute to - namely: Terraform, Kubernetes, Docker, Helm and countless small components of that eco-system? Can I truly wake up tomorrow and tell myself - that's it, I'm done, no more Go in my life?!

With great power - comes great responsibility. Yes, maintainers and owners, legally speaking, don't own anything to anyone. It is right there in the license. But think again, as a maintainer, when you decide to share your work with the public, outside of the court ow law - do you really not feel like by doing that, you embark on some additional responsibility in your life? Isn't that, after all, the reason we all are here on this earth - to take on some personal responsibility?

"You become responsible, forever, for what you have tamed." - Antoine de Saint-Exupéry

Lastly, let me close with this. This culture of unacceptance of any negative feedback is sickening. How do you become a master? Well, first - you got to risk to become a fool. Because no one in the world can learn and become a master overnight, without making some mistakes. How would you know you've made mistakes, if no one tells you? No one likes to receive negative feedback, and a rare individual willing to give it. That's why the ones that are - are extremely valuable. They are your only chance to become a master. But in this day and age - nobody can be wrong, nobody can be offended, nobody can make caustic comments. We, as a society, voluntarily decided to deprive ourselves of this very important socioneurological self-regulation mechanism. Because if you are already perfect - what does that tell you about who you should become? Or is it like - you are already done with your character development? That's it? Life's over? So yeah - speaking of under appreciated work - I think I am topping it here together with @amery and a few others. You're welcome. I know, as (more or less) a grown up human, who can discuss in a logical way - you will (maybe - in time) appreciate my effort to provide you with this wake-up call.

Answer 62 · 2023-04-11T14:58:47.000Z

Every time I try to revive this topic it triggers a wave of off-topics, and the actual attempt to revive it gets lost in between. Please, let's keep the discussion on point! This thread already got quite messy.

Mentioning @ianlancetaylor (not sure if there's a more suitable person to mention). Please provide some guidance if you can, about what should be done for this proposal to be considered, and to at least become an "Active" one in the Proposals Project. Should I update the original message to clarify the actual proposed solution? Would that help?

Answer 63 · 2023-04-11T15:16:35.000Z

@burdiyan, the main blocker for considering this from my perspective is the migration path for the proxy.

If we want to change the contents that the proxy serves for a given repo×version, we need some mechanism to avoid spurious checksum errors while the ecosystem has a mix of different versions of the go command. (See previously #30369.)

If we don't have a concrete plan for that migration, it's just not feasible to consider changes. If we do have a concrete plan for that migration, then we can talk about how costly it would be to implement, both in terms of effort from the Go team and churn for Go users in general. That cost is likely to be a very important factor.

In the meantime, note that you can always run your own server to convert your repo from a raw Git repository to the GOPROXY protocol with whatever transformations you like, and point the rest of the world to that repo by serving a go-import directive as described in https://go.dev/ref/mod#serving-from-proxy.

Answer 64 · 2023-04-11T16:33:29.000Z

I have put this on the list for consideration by the proposal review committee (I did that a few days ago).

But the committee seeks consensus, and I don't know whether we have that. The problem, as I understand it, boils down to "in some cases, the module under development contains directories with files that the go command should ignore." Often this is primarily a matter of efficiency, but in some cases those directories wind up containing Go files that are not actually part of the module.

In #30058 we agreed that now that we use modules we can fix the problem by putting an empty go.mod file in directories that we want to ignore. However, that is imperfect because it requires adding the file to directories that are created and removed by other tools.

We've discussed adding a ignore directive to the go.work file. That seems to me like a clean place to put a directive, because many of these directories (node_modules, bazel-out) are features of the build system, not of the source code as such. The go.work file is conventionally not committed to a repository, and as such the set of ignore directives has to be recreated by each user. If we permit the ignore directive to refer to a separate file containing the list of directories to ignore, this seems to me to be not so bad: a small one-time cost for each developer. It would be helpful to hear about other downsides to this approach.

We've discussed adding an environment variable with a list of directory patterns to ignore. As with the go.work file, this requires a one-time setup by the developer. For a set of developers, it's straightforward to provide a default env file that they have to put in the right place (on Linux, $HOME/.config/go/env). It would be helpful to hear about downsides to this approach.

We've discussed adding a go.ignore or .goignore file that would be committed to a repository. As @bcmills mentions this has a transition problem. For directories like bazel-out, it's also a confusion between the module and the build system being used. Many modules are built by different build systems. Should each module have to carry a go.ignore file for build systems that the module developer doesn't even use? That doesn't make a great deal of sense. Perhaps it should only be in the main module, whatever that is? But that leads us back toward a go.work solution, as that is where we describe things for the main module.

Answer 65 · 2023-04-11T17:12:53.000Z

being outside version control (go.work) breaks reproducibility, go would do different things in one environment or another.

go.ignore on the same directory as the go.mod is the most authoritative place regarding knowledge of the structure beneath. And it can't be recursive because it won't be visible when someone imports a submodule

Answer 66 · 2023-04-11T17:50:10.000Z

The ignore file needs to be able to be committed to the repo, asking every user to create a go.work file is definitely worse / not viable.

Look at .gitignore - it works this way. Commit a file ignoring some paths. Why can't go do this too? There is a problem of course with newer versions recognizing the ignore file and older ones not. But this seems like an OK tradeoff when compared with the massive downside of using a file not committed to source control.

Answer 67 · 2023-04-11T19:17:41.000Z

@amery @paralin Please think about the case of directories that are aspects of the build system that we want the go command to ignore, such as node_modules and bazel-out. Is the correct answer for every module to add those directories to a go.ignore file? I don't think that is a good answer.

It's true that git has a .gitignore file. It's also true that git has a core.excludesFile configuration variable. Does Go need an equivalent to both of these? I don't see why the former is the right answer in all cases.

What are examples of directories that the go command should ignore that are not part of the build system?

Answer 68 · 2023-04-11T19:24:47.000Z

@amery @paralin Please think about the case of directories that are aspects of the build system that we want the go command to ignore, such as node_modules and bazel-out. Is the correct answer for every module to add those directories to a go.ignore file? I don't think that is a good answer.

@ianlancetaylor We don't need to ignore every build system on every module, each module maintainer knows exactly what to ignore if anything. the entry gets committed once on the module that needs it and that's it. Sometimes they are "well known", sometimes they are project specific, but when they happen below the directory with the go.mod we know they happen and we can perfectly narrow the rules and commit it. For that particular module, not all of the them. No magic required, just a mechanism to tell the tooling to ignore a particular directory or file

Answer 69 · 2023-04-11T19:50:47.000Z

@bcmills I find it weird/unfortunate that a "main blocker" for this is the Go proxy, given that the modules where this feature is most needed tend to be ones that will never be transmitted across a proxy anyway.

Or to put it another way: I don't particularly care whether these ignored files are included in the proxy or the module checksum. I just want them to be ignored by the Go tool to the extent that they cannot (for instance) make go list slow. And more importantly, I want the ignore mechanism to exist so that it can be used by other tools that need to process my local module code so that they don't all need their own bespoke mechanisms (like .goimportsignore and gopls's directoryFilters). Keep in mind that, as the PR title says, this is about the ecosystem.

Riffing on that a bit, ISTM that we could even start by adding go.ignore purely as a convention (supported by goimports and gopls but not used by the Go tool at all) and then look to selectively incorporate it into the Go tool in some way later on. For instance, go list could run in a mode where it respects go.ignore and tools that use go list would be encouraged to use that mode.

In the meantime, note that you can always run your own server to convert your repo from a raw Git repository to the GOPROXY protocol with whatever transformations you like, and point the rest of the world to that repo by serving a go-import directive as described in https://go.dev/ref/mod#serving-from-proxy.

I don't know how relevant this is. As I understand it (and certainly for me), the problems that require an ignore mechanism are mostly about local development of modules (that you probably obtained with git clone or something). It's not about how the module appears to others who obtained it via go get or whatever.

Answer 70 · 2023-04-11T19:57:30.000Z

I'm getting the impression you believe the build systems get into pkg/mod and mess with third-party packages.
This is not the case, it's our repositories, with our build systems, grouped by domain instead of language.
we don't need to inject exclusions on every existing module, we don't need to inject exclusions on any module from someone else. Only on those where we know as a fact and to some degree by choice, different systems overlay

Answer 71 · 2023-04-11T19:59:52.000Z

@amery

each module maintainer knows exactly what to ignore if anything.

I guess I don't understand that. If I write a module how can I know how people will build it?

If this only applies to modules that are completely under your control that nobody else will build, then I don't think the cost of a one-time setup of a go.work file or an environment variable is an intolerably heavy price.

Answer 72 · 2023-04-11T20:07:15.000Z

@ianlancetaylor

I guess I don't understand that. If I write a module how can I know how people will build it?

you don't need to. your module will be safe in pkg/mod as a dependency. this only affects complex projects where different languages cohabitate in the same repository

the reason to not use go.work or environment variables is because those methods are personal and they WILL vary from developer to developer, but all those developers or CI pipelines will clone the repository and the tooling will find the ignore files instructing them what to skip

Answer 73 · 2023-04-11T20:19:39.000Z

I guess I don't understand that. If I write a module how can I know how people will build it?

That is the problem of a tunnel vision I was talking about - it appears to me, that your assumption is, that every repository with Go code is automatically a Go module. That is not the case. A repository might be a mix of different technologies (in an example of a mono-repo), that just happens to include some Go code among other things. Users will want to build that repository as a top level of their hierarchy, but I don't see this repository to be used as a dependency anywhere else (otherwise reusable code will be just moved to a standalone repository manifesting a Go module).

The reason this is still a source code issue and not a build system issue - you may be having hundreds of people working on that mono-repo. The fact that nothing else depend on this repository doesn't automatically characterize the scale of it to some small little scope only used by single build system and only a few people. It is not feasible nor reasonable to expect hundreds of developers to have to configure their local environment the same way to be able to develop locally. Reminds me of times where I had to checkout code from perforce to a certain location on a disk D:, otherwise it would not compile.

Answer 74 · 2023-04-11T20:27:26.000Z

Also, there is an existing precedence of this - if I use replace directive in my go.mod file today, I will not be able to use that module from somewhere else as a dependency. So - this behavior wouldn't be drastically new either.

Answer 75 · 2023-04-11T21:07:04.000Z

@bcmills I don't fully understand how the Go proxy becomes the main blocker for this. Can you please elaborate and provide a link to the problem details? (sorry, if it was discussed before... this was this thread got too long :-()

To me, this proposal is to come up with a unified, standardized mechanism instead of all different mechanisms developed by individual tools in ad-hoc ways. Gopls could use directoryFilter without go module proxy's modification or any complex migration plan. So, I want to know why module proxy becomes relevant for this proposal.

(EDIT: I guess this is related to compute the checksum of a module. However, given that the motivation of this proposal is to support tools like gopls and analyzers used during development, IMO this ignore mechanism shouldn't be used in checksum computation or decide what to be included in the module zip file. That means at some point the go command still needs to scan and read all files if the repo is a module.)

@ianlancetaylor Ability to check in certain editor or dev tool configuration is getting important. For example, when a user opens a repo from GH Codespace or Google cloud workstation or web ide, they want the environment to be immediately available for development. Requiring to set up go.work or an extra step to prevent gopls or lint tools from excessive resource usage isn't ideal. If go.work is the way to configure the shared environment, I think we need to stop discouraging check-in of go.work.

@antichris suggested GOIGNOREFILE in #42965 (comment) I expect the go.ignore file to allow flexible patterns like .gitignore or other dot ignore files. For web ides or common dev env setup, users may have an option to configure the env vars (e.g. .vscode/settings.json, .env, ...) and check in the configuration. Is it still under consideration?

And, finally, I hope we stop recommending a dummy go.mod file. That adds extra complexity to the module proxy implementation and the hosting sites.

Answer 76 · 2023-04-11T21:24:56.000Z

@hyangah Thanks for pointing out GOIGNOREFILE. I don't see any difficulties with that. But it would seem to face the same objections as an environment variable with a list of patterns to ignore.

Answer 77 · 2023-04-11T21:38:06.000Z

@ianlancetaylor I wish we could find a solution for the per-workspace go.env idea (discussed in #57179 and other places). At least, I believe checking in .env or editor settings to configure the env var is an option if sharing certain env vars among project members is a concern (these files are outside of the Go project's scope :-))

I hope GOIGNOREFILE and file is easier to read/use than an env var listing ignore patterns. Maybe a tired user can have go env -w GOIGNOREFILE=go.ignore for which the go command walks up the directory tree as it does with go.mod or go.work search and if go.ignore exists, apply the rule.

Answer 78 · 2023-04-11T21:52:37.000Z

@hyangah I don't yet see a problem with a per-workspace go.env file. The problems are with a set of per-module go.env files.

Answer 79 · 2023-04-11T22:10:27.000Z

To solve the version transition issue, the go compiler could use the language version in go.mod. If the version is say 1.21 or higher, use the .goignore or go.ignore. If not, ignore it.

...such as node_modules and bazel-out. Is the correct answer for every module to add those directories to a go.ignore file? I don't think that is a good answer

It's a separate problem to ignore every possible compiler temporary directory vs. just ignoring things that appear in the working directory while developing the program. This would be used in cases where the module developer has a package.json next to go.mod and needs to ignore node_modules with the knowledge that this directory will be present.

Answer 80 · 2023-04-11T22:34:31.000Z

@ianlancetaylor The main concern around go.env ~~per-workspace~~ per-work module was probably security (#57179 (comment) #57179 (comment)). On the other hand, some popular editors have implemented trusted workspace features, so I guess this can be mitigated through UX/UI work.

I am not quite sure what you mean by "a set of per-module go.env". I thought the original proposal in #57179 (comment) meant only the go.env from the work module (or next to the applied go.work) will be used, but it's possible I misunderstood the proposal. Given that the env is often about the local environment users are running (same as $GOROOT/go.env or user profile go.env), I think it makes little sense to attempt to use go.env from dependencies. If there are some env var that should affect build of a individual dependency, I think that probably belongs to go.mod.

Answer 81 · 2023-04-11T22:36:53.000Z

It's also possible that I misunderstood.

Answer 82 · 2023-04-12T07:24:47.000Z

That is the problem of a tunnel vision I was talking about - it appears to me, that your assumption is, that every repository with Go code is automatically a Go module. That is not the case. A repository might be a mix of different technologies (in an example of a mono-repo), that just happens to include some Go code among other things. Users will want to build that repository as a top level of their hierarchy, but I don't see this repository to be used as a dependency anywhere else (otherwise reusable code will be just moved to a standalone repository manifesting a Go module).

I do think everything is said here; IMO, we should stick to something simple like a .goignore file working the same way .gitignore is working. We've all been using them without any pb knowing what we were doing. In that case, and especially for a mono-repo where different languages are used, this file might simply tell the Go compiler not to embed useless files on the module. Haven't you seen examples of Go module that carry Javascript or Python code because of the structure of the mono-repo?

Answer 83 · 2023-04-12T15:46:20.000Z

It looks like two different problems are discussed in this thread.

making tools used during development (more aligned with the use case of go.work) ignore some directories/files. For example, temporary directories created by other language build systems, or a large number of files that aren't necessary to build/analyze the go project. That was one of the main motivation in this proposal I thought.
making the go command ignore some directories when the go command is packaging a go module from source code. That is the cause of the transition issue the Go command team is worried about. As @paralin pointed out, the upcoming change in the meaning of the go version (#57001) may help if we carefully coordinate the module proxy's go version upgrade and also adjust the go command to check the checksum later.

However, I still wonder using one ignore list for these two problems is the right choice.
I see the ignore pattern lists for 1) and 2) may look often identical but not always the same.

For 2), I see some projects developed workarounds such as

organize the source tree so they can place go.mod to create a desired module boundary. That's sometimes tricky and we also know multi-module repo is still hard to work with for both development (e.g. go test ./... doesn't work), release (e.g. releasing interdependent packages in correct order is not a trivial task), and product branding.
clone/copy only the go related part to a separate repo (e.g. k8s, dagger) and let go module get distributed from there, or run a mod protocol server (that's what @bcmills suggested in this comment). I wonder if there could be a toolset to help people who want to have separation between the source code repository and the module distribution source.

Answer 84 · 2023-04-12T15:52:47.000Z

organize the source tree so they can place go.mod to create a desired module boundary.

please also consider that is generally accepted as a good practice to arrange code by domain, not language.

it's worth mentioning that while switching to using go.work I ended being forced to add go.mod and a non-empty .go at the root of the repo otherwise go list gets upset.

Answer 85 · 2023-04-12T18:00:22.000Z

I initially didn't think that the module proxy would be relevant to my proposal, but I guess it is a valid concern. My main motivation for the proposal was to address local development issues. I'd personally be OK if module proxy and packaging would completely ignore the existence of the go.ignore (or whatever it ends up) file.

Although I would be happy if go.work could support this, I strongly believe (as some other people here) that the ignore file should be included in the repository so that team members and CI can easily share it. Asking everyone to replicate the ignore patterns on their machines each time is impractical, especially when these could change over time.

I still don't understand why there's such a strong argument for not including go.work into the source control. Many other programming languages have some way to specify a multi-module workspace by having some config file in the repository root, similar to go.work. E.g. Rust allows to specify nested crates in the main Cargo.toml, JavaScript does the same in package.json or pnpm-workspace.yaml. And these files are expected to be in the source control. Personally, most of the time I'd want go.work to be in the repository as well, except only one case: when hacking on some dependency that I've "forked" into some local directory.

So, given all that, I'd prefer specifying ignore patterns in a separate file (personally I like go.ignore better than other alternatives). And whether this file is taken into account or not for module proxy: I have no opinion here, as I'm not aware enough about the details of the module proxy.

Answer 86 · 2023-04-12T18:04:45.000Z

I still don't understand why there's such a strong argument for not including go.work into the source control. Many other programming languages have some way to specify a multi-module workspace by having some config file in the repository root, similar to go.work

@burdiyan does go.work support nesting? if they are checked in people working on two potentially independent projects might need to open modules from different repositories each potentially with their own go.work

Answer 87 · 2023-04-12T20:33:07.000Z

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Answer 88 · 2023-05-03T17:10:50.000Z

As a note, the forward compatibility work in #57001 does make it possible for us to change the set of files included in a module based on the go line in the go.mod file, so there are some new doors open here that weren't before.

Answer 89 · 2023-05-10T21:47:26.000Z

Talked to @bcmills and @matloob about this. We think this should be possible to do after Go 1.21's forward compatibility work (#57001) is released. That is, it couldn't happen before Go 1.22. But at that point we think it would be OK to add a new ignore directive in go.mod that would only take effect in go.mod files that say go 1.22 or later. The ignore directive would take a directory name to ignore (also ignoring all directories within it, recursively):

ignore ./node_modules

Just like any directive, it can be factored:

ignore (
    ./node_modules
    ./obj
)

The leading ./ is necessary to make clear to readers that this is a directory and not a module path, same reason use requires the leading ./ in go.work.

If you ignore ./x there is no way to un-ignore ./x/y or ./x/y/z.

Are there any concerns with this design? Would it fail to address any needs that have been raised on this issue? Thanks.

Answer 90 · 2023-05-10T22:17:24.000Z

Are there any concerns with this design? Would it fail to address any needs that have been raised on this issue? Thanks.

It does sound like a good minimal viable first step that can be implemented performantly.

My only concern is that if other tools like code formatters (mvdan/gofumpt#250) make use of the mechanism, they would also need a ignore mechanism for files so they can for example exclude some .go files from formatting. I guess in such a case, the tool would still need to provide its own ignore mechanism.

Answer 91 · 2023-05-10T23:51:55.000Z

@silverwind In this case I think that's fine: we need a way to tell the Go compiler "skip this directory," other tools like gofumpt could as you say, also have a "ignore" configuration flag.

Answer 92 · 2023-05-11T01:12:20.000Z

Are there any concerns with this design? Would it fail to address any needs that have been raised on this issue? Thanks.

It does sound like a good minimal viable first step that can be implemented performantly.

My only concern is that if other tools like code formatters (mvdan/gofumpt#250) make use of the mechanism, they would also need a ignore mechanism for files so they can for example exclude some .go files from formatting. I guess in such a case, the tool would still need to provide its own ignore mechanism.

I don't use gofumpt but I do believe the idiomatic way of excluding specific files is to use a comment flag on that file following //go:build fashion

Answer 93 · 2023-05-11T08:27:25.000Z

Agree, there are ways in gofumpt, and it's kind of off-topic here.

The crucial part regarding tools like gofumpt or golangci-lint is that they discover go files via go's built-in mechanisms which will be subject to the new ignore list. With it, they can quickly skip over node_modules without trying to discover .go files in it (which has been an issue in the past as some node modules ship .go files).

Answer 94 · 2023-05-11T12:37:58.000Z

@rsc @bcmills @matloob Thanks for reviewing this topic.

■ Is it possible to consider more flexible filter rules than the list of directories to exclude?

I am primarily interested in using this feature to help tools relying on go list ./… share filtering rules. Though not as powerful as .gitignore or other ignore mechanism, gopls's directoryFilters offers more flexibility than what’s presented in #42965 (comment).

The gopls filter allows both "exclude" and “include” rules (+ vs -). I think that’s useful when working in a multi-language project with many directories and some of them are using go.
The gopls filter allows directory matching at arbitrary depth (**). We didn't have this initially, but over time, we learned this is useful over time, in particular, in large complex multi-language projects or when people move around code for refactoring. #46438.

■ Can we clarify exactly when this ‘ignore’ rule will be used?

As summarized in #42965 (comment) I observed two similar but different feature requests discussed in this thread. Is this one ‘ignore’ list intended to be applied to the both problems?

(1) want to limit the directories/files that gopls, go mod tidy and tools like linters operate on. This is mostly the target go list or x/tools/go/packages can operate on, and similar to directories named like testdata, or .*, or _*.
(2) want to limit what's included in the module zip file. This is mostly the target go mod download can operate on. AFAIK this is a new functionality - e.g. go mod download still includes testdata, .*, etc. in the final zip file. Related feature requests (closed in favor of this proposal) are: #50225 #52807

According to the feature request camp (2), some examples users want to exclude from their module zip include some large test data (so tests that reference the excluded data) or CI/CD or dev tools. The directories may include valid Go packages or related assets needed during development, so they shouldn’t be excluded from go build, gopls, or other tools.

The opposite is also possible - files can be included in the module zip but users don’t want their go tools to care about them during development (e.g. directories containing large number of assets, mono-repo structured based on ownership, …) - but I think that is less convincing.

■ Lastly, I personally prefer having a separate file for this to make the file easier to inspect (go.mod grew already large in some projects, not as large as NodeJS's package.json) but I think we can live with a long go.mod.

Answer 95 · 2023-05-11T20:54:24.000Z

Is it possible to consider more flexible filter rules than the list of directories to exclude?

I think we could plausibly consider supporting Go package patterns (particularly ...). It's not clear to me that they're worth the complexity, though, given that one could also write a third-party tool to automate updates to a simpler list (perhaps even using go generate to maintain it).

Answer 96 · 2023-05-11T20:54:33.000Z

Can we clarify exactly when this ‘ignore’ rule will be used?

IMO it should function like _ or testdata for matching package import patterns (the directories would be filtered out of wildcards, but could still be referred to by explicit paths), and should be excluded from the module zip file as well.

Answer 97 · 2023-05-11T20:57:43.000Z

Lastly, I personally prefer having a separate file for this to make the file easier to inspect (go.mod grew already large in some projects, not as large as NodeJS's package.json) but I think we can live with a long go.mod.

That's fair. In particular, the ignore information is not needed at all during dependency resolution.

Answer 98 · 2023-05-11T20:59:10.000Z

The gopls filter allows directory matching at arbitrary depth (**). We didn't have this initially, but over time, we learned this is useful over time, in particular, in large complex multi-language projects or when people move around code for refactoring

One major difference to note is that gopls configuration is typically system-wide, not workspace wide. This new configuration would be relative to a specific module (and would be versioned). That alleviates much of the need for wildcards.

Answer 99 · 2023-05-11T22:23:21.000Z

@rsc

Are there any concerns with this design? Would it fail to address any needs that have been raised on this issue? Thanks.

Yes -- for my needs, I need to express arbitrary nesting. For example, I want to ignore any node_modules directory at any depth.

Answer 100 · 2023-05-11T23:20:35.000Z

@rsc

Are there any concerns with this design? Would it fail to address any needs that have been raised on this issue? Thanks.

Yes -- for my needs, I need to express arbitrary nesting. For example, I want to ignore any node_modules directory at any depth.

Maybe something like ignore ./**/node_modules?