Update publicsuffix dependency

Hi,
wanted to confirm what is the current version of public suffix list that is being used by the authority for certificates limits issuance. Does it include this commit publicsuffix/list@9636089 ?

It does not, yet.

Can we update it? Can you point me to the last commit in which it was updated? I can see only this issue: #1251

Yep, that's the last update commit. Note: we normally update from golang/net's upstream, however, it may be easier to start doing updates directly in our own fork at https://github.com/letsencrypt/net. We're a bit short-handed this week, but I'll treat this as a request to update it for next week when more of my colleagues are around to review. Thanks!

Thanks, one more question: how long does it take to then release the new version of the CA (or to consume by running version the updated config)?

Usually we deploy about once a week, barring other issues.

Hi Jacob, any update on this, can I help you with update of suffix list?

@kuba for helping to resolve this.

I know you must feel frustrated but we have lots of constraints and demands on our time. Please have some empathy for the team.

You heard what our deploy schedule was. We'll be doing a deploy and you will get an update when we do.

Pulling unrelated folks into this discussion isn't going to make things move faster. There isn't anything they'll be able to do to change our deploy schedule. It'll just distract them from the other work they can do.

To be more clear, you can check out https://letsencrypt.status.io/pages/history/55957a99e800baa4470002da for when deploys go out.

I assumed there need to be a code change done in order to pull new version of public suffix list, isn't that the case?
If not, then I am happy to wait for next release. (It is not about "frustration" - not at all.)
If this is an automated change that I could possibly perform then happy to do that. If it doesn't require a code change then let it roll.

Problem here is that upstream https://go.googlesource.com/net/+log/master/publicsuffix updated table.go around 6 weeks ago, so publicsuffix/list@9636089 could not possibly be included in it. @mbwalas, you need to escalate to Go team and make sure they update that package. Once it happens Boulder team will need to merge https://github.com/letsencrypt/net/tree/master/publicsuffix with upstream, then

boulder/Godeps/Godeps.json

Line 116 in 3d7413a

"ImportPath": "github.com/letsencrypt/net/publicsuffix",

will need to be updated, then code will be rolled out to staging, then to production. In short: I still think there is around 1 or 2 weeks of waiting until this will be deployed anywhere, at least.

@jmhodges, not so random folk + I hope above helps to understand the situation better ;) Somewhere lost in translation was that particular PSL commit we're after, hence a bit of misunderstanding in the discussion. Also, do we happen to have any fast lane mode that circumvents the whole route I described above?

Nope, we don't. That's the route. Thanks!

I sent out https://golang.org/cl/18724 to update Go's publicsuffix list. Waiting on a review.

Done ^^

@jmhodges I'm both a PSL committer and a Go user, and I'd like to contribute. As Go embeds the PSL in the source itself, I'm planning to provide periodic updates to the Go repo (assuming that also is fine for the Go team).

I mailed a PR yesterday for review with the latest changes, and cced @bradfitz
https://go-review.googlesource.com/#/c/19410/

What is the best way for me to contribute the patches to this repo? Do you want me to provide separate updates via single PRs, or do you prefer to keep this repo in sync with /x/net/?

I'm currently automating the process of validating, linting and testing the PSL, and I'll also explore some kind of automatic packaging into a periodic Go-repo update. I'm open to requests and feedback.

I'm a little concerned by regular updates to this huge blob in the golang.org/x/net git repo. It adds about 0.1MB to our history per commit, which isn't much, but would add up if we're doing this regularly.

It's also annoying to review these, since I basically have to re-do the CL whenever somebody sends it to me to verify it hasn't been tampered with: I have to at least cherry-pick it in, then update the test files (which at least doesn't take 10 minutes like the other file), and then run the tests.

I'm wondering if we should add a different mode to Go's publicsuffix package where instead of baking in the data structure, it instead regularly polls or otherwise subscribes to the master list and keeps it all in memory (in a larger format, not the 10 minutes of CPU "crushed" form). Then boulder would always be up-to-date with what's live in the public suffix list, without needing to wait for Go.

/cc @nigeltao

Thanks for the offer @weppos! And your concerns about frequent updates in golang.org/x/net make sense, @bradfitz. Let's try and find a better way. I opened #1479 to discuss alternatives.

It may be worth having a golang.org/x/net/publicsuffix/parser package that can build a (not as optimized) table at runtime instead of at code-gen time. I don't really have time to work on it any time soon, but I'm happy to review any proposals and API designs.

In any case, AFAIK letsencrypt doesn't have to use the golang.org/x/net/publicsuffix package. You could simply fork the package, with a note that your flavor is simply updated more frequently than the upstream flavor. The code-gen script should be easy to run yourself at whatever cadence you like.

Closing in favor of #1479.