ipfs/boxo

[WIP] Frequently Asked Questions about this project

BigLep opened this issue · 1 comments

BigLep commented

See #190 (comment) for latest

vvv Not reviewed or up-to-date vvv

Table of Contents

Purpose

Using this as a quick way to add FAQ entries / notes while there is a lot of transition activity around this repo. In future, these will get moved to durable markdown files, but for now allowing edits here to reduce any friction with capturing this content.

Rational FAQs

TODO: 2023-03-30 this needs to be expanded, including content from other issues and slack threads.

What are the reasons/goals for doing repo consolidation?

We want to make it easier to contribute to PL's Go IPFS codebase. Why?

  • The IPFS Stewards, the maintainers of most IPFS org repos, are shifting focus to libraries.
    • Provide a first-class library experience, to enable users to build their own binaries/implementations instead of piling everything into Kubo
    • Existing libraries are hit-or-miss, with large amounts of tech debt (poor testing, unsafe APIs, multiple versions/forks, half-completed migrations, "experimental" features sitting unfinished, etc.). We want to refactor these but refactoring is very time consuming given the large number of repos and versions involved.
    • Library components should be tested together
  • We want to grow the Go IPFS community to include more than just PL engineers.
    • The large number of repos + versions is intimidating to newcomers
      • It is difficult to understand the impact of a change across all the repos+versions
      • There is no cross-repo testing feedback until you plumb everything up to Kubo

Why is the Boxo project consolidating repos and using example-driven development despite ecosystem friction?

We believe that the current go ipfs ecosystem is suboptimal due to issues like incompatible versions, complex APIs, and hard-to-navigate projects. We think a shared monorepo following an example-driven development will create a better
ecosystem. Although the transition causes friction, we developed a migration tool to help with migration. We believe the tradeoff is worthwhile for long-term ecosystem improvements, but the validation of this approach won't come until evaluating metrics and user anecdotes 3-6 months after.

Why do maintainers care about reducing the number of repos they maintain?

Repo maintenance costs include:

  • Keeping dependencies up-to-date
    • This is non-trivial as it often requires chasing down other dependencies in the dependency graph...mostly we don't do this until we have to
  • Releasing new versions as necessary
  • Making sure CI is still working
  • Migrating from Travis/CircleCI to Actions (still in progress)
  • Rolling out unified CI
  • Backporting changes across major versions as necessary
  • Manually testing impact of new code changes on downstream consumers
  • Monitoring issue trackers, PRs, etc.
  • Updating submodules
    • Commonly used for testing, example code, etc.
    • Often these contain circular module dependencies which complicate propagating breaking changes

What alternatives were considered and why were they ruled against?

Just focus on docs/examples in a repo with a known set of versions that play nice together

Given this isn't free for maintainers or those affected, what has been the cost/benefit analysis?

What are some concrete usecases where repo consolidation would have been beneficial?

2023-03-02 note: this will get extended in the next week. @BigLep only seeded with a few things.
Below are anecdotes from the last few months where EngRes IPFS Stewards wished for directly or saw need from others for consolidated repos:

  • Changing the fetcher API - @aschmahmann to fill in. @BigLep jotted this down during 2023-02-28 conversation.
  • Finding a bug in a sub-repo and not being able to handle
  • 2023-02-26 Bacallhau raising concerns about their dependency tree.
    https://filecoinproject.slack.com/archives/C02RLM3JHUY/p1677409745784779
  • 202302 security event handling where doing a fix spanned 3 repos instead of the ideal one. This mean 3 security advisories, releases, etc.
  • 202302 Gala Games - After experiencing Kubo footguns that affected DHT performance, they raised their hands to be consumers of a project like boxo so they could assemble a IPFS implementation tailored to their needs.
  • In 2022 IPFS Stewards spent months plumbing contexts through 40+ repos, some having multiple incompatible versions. The complexity involved is quite high--you need to find an appropriate topological order of the dependency graph, make the changes & test & release repo versions incrementally, etc. And if you make a single mistake, you have to start over.
  • 202303: #242 . This would have needed to be a complex DAG of 4 PR and go.mod bubbling for what is a simple API extension, this is now 1 PR. Those improvements would take an hour before, this was 5 minutes of a maintainer's time.
  • 2023-03-31: Unsolicited feedback from engineers working on ipfs/gateway-conformance-testing: "❤️ the boxo consolidation. In conformance we went from importing ~8 packages to open a car file containing a unixfs node to 4 packages. The lib is easier to work with, it feels more consistent, and it’s less overwhelming to get started."

Copied or Migrated Repos FAQs

The FAQs below concern all the repos that were copied or migrated into boxo to bootstrap the library. Some of the repos are copies of repos that are maintained by others, but most are now not maintained.

What repos have been copied into Boxo?

The authoritative list is in https://github.com/ipfs/boxo/blob/main/cmd/migrate/internal/config.go. Many of these were handled as part of #202.

What will happen if there is a security issue in one of the legacy ipfs/* repos that has been copied into boxo?

@ipfs/kubo-maintainers (which primarily maps to PL EngRes IPFS Stewards as of 202203) will certainly handle patching boxo. If there are maintainers for the original/source repos, they will need to handle patching/disclosing. @ipfs/kubo-maintainers will certainly coordinate and share work, but won't handle communication with or updating of the affected consumers of the original repos that boxo copied from. If there are no maintainers for the original/source repos, there will likely be internal private PL EngRes chats analyzing which projects are impacted, and it will then be up to those projects to determine whether they want to patch the existing repos or update to use boxo. (We understand and expect that the "update to boxo" option will often not be possible under tight timelines given boxo's plans to upgrade its dependencies frequently and refactor the existing code.)

I don't like the "deprecated" warnings in "not maintained" repos that have been moved to boxo. What can be done about this?

The intent of the "deprecated" warnings was to be clear to consumers about the status of the repo: that it isn't maintained that there is a repo where similar code is being maintained. If another maintainer comes along, then they can follow claim ownership and the state of the repo can be adjusted. This satisfies the requirement about being clear to users about who owns and its status.

How does one claim ownership of a "not maintained" repo?

A new maintainer can

  1. create a PR that includes:
    • Revert of the deprecation types
    • Removal of the "not maintained" notice from the README
    • Addition of themselves to CODEOWNERS
  2. Get @ipfs/kubo-maintainers attention by @mentioning them in the PR and/or posting in the venue(s) discussed in https://github.com/ipfs/boxo#help

Assuming the new maintainer is known/trusted, @ipfs/kubo-maintainers will merge.

Will the "not maintained" ipfs/* repos be left around to rot?

PL EngRes @ipfs/kubo-maintainers don't want to leave a wasteland of unmaintained repos. We'll come back through and archive repos that haven't had a maintainer step up in 2023Q3. See #201

Logistics FAQs

What is the release process for boxo?

At least as of 2023-03-02 this is not well defined. This will be resolved in #170

What is the response time for issues or PRs filed?

What are some projects that depend on this project?

The exhaustive list is https://github.com/ipfs/boxo/network/dependents. Some notable projects include:

  1. ipfs/kubo
  2. filecoin-project/lotus
  3. protocol/bifrost-gateway
  4. ipfs-shipyard/ipfs-check

Why is this named Boxo?

TODO: extract portions from #215

Good sources of information for more FAQ questions/answers that should get pulled in here

  1. ipfs/kubo#8543
  2. #boxo-maintainers thread: https://filecoinproject.slack.com/archives/C04M8232QRW/p1677103751480109
guseggert commented

These have mostly gone into the wiki here: https://github.com/ipfs/boxo/wiki

What wasn't put in the wiki has been added to README.md and RELEASE.md