ipfs/in-web-browsers

Verifiable HTTP Gateway Responses

Closed this issue · 2 comments

lidel commented

Summary

Various organizations are exposing IPFS content via Public HTTP Gateways (eg. Cloudflare).

There are users who can't run local IPFS node, but still could get some guarantees provided by content-addressing if they have software capable of re-calculating CIDs from arbitratry payloads.

This issue tracks ways we could do that.

Problem statement

  • It is not enough to "fetch data from gateway and calculate CID", as we don't know how DAG looked like, so if CID does not match, we don't know if it is false-negative.

Use Cases

  • (UC1) Package Manager fetching archive from HTTP Gateway
  • (UC2) User running ipfs-companion or other browser extension
  • (UC3) IoT / low power devices that are unable to use true p2p, but can reach HTTP gateways
  • (UC4) Web browser that wants to natively delegate IPFS handling to public gateways, but provide users with integrity guarantees of IPFS

Work

  • Research limitations of existing vendor-specific prior art
  • Research how to resolve CID of an arbitrary path and get insight into its DAG structure
    • A: 💢 additional lookup via local js-ipfs used only for that or fallback to http://ipfs.io/api/v0/
    • B: 💛 Gateway exposes all metadata required for Reproducible File Import via HTTP header, so no additional lookup is required
      • Gateway support for CAR export (#170) is a variant of this
    • C: 💛 UnixFS files identified using hash of the full content (protocol/beyond-bitswap#29)
      • Improves on top of B because it does not require client to know/implement Unixfs/IPLD specifics, just read hash function type from CID and then compare it against the hash of entire file.
      • Requires separate provider records, so won't work for legacy data/links
  • Gateway can return all parameters needed for re-building DAG during validation (or CID can be validated directly)
    • BLOCKED by ipld/legacy-unixfs-v2#15
    • Note: this may become not a problem if we switch approach to CAR import/export: #170
      • if we are able to control the depth of CAR export, we could have parallel streaming from multiple gateways
    • Note: alternative to CAR is RFC|BB|L2-08: Handle Arbitrary Block Sizes
      • This is better than CAR, as does not require user agent to support CAR format and can verify regular responses based on URI alone
  • (UC1)(UC4) CLI tool is able to fetch HTTP payload and validate it against a CID in offline mode without need for a full IPFS node
    • Highly useful for use in Package Managers
  • (UC2) Browser Extension is able to validate HTTP payload against its CID
  • (UC3)(UC4) full DAG can be fetched from public gateway
    • CAR export built-in feature of HTTP gateways: #170

Resources

Prior Art

  • (UC2) Cloudflare

  • https://tools.ietf.org/html/rfc6249: Metalink/HTTP: Mirrors and Hashes
    Very relevant RFC, which provides HTTP-native semantics for returning Digest with the payload:

     Link: <http://example.com/example.ext.asc>; rel=describedby;
 type="application/pgp-signature"
 Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO
 DYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ==

IPFS

lidel commented
  • done: block and CAR response formats (ipfs/kubo#8758) are supported by HTTP Gateway provided by go-ipfs 0.13+
  • wip: specs for "Trustless Gateway" are part of HTTP Gateway specs ipfs/specs#283
  • next: spec for "Trustless HTTP Clients" needs to be created, documenting full request cycle and minimal subset of IPFS required
  • 🎉1
lidel commented

This is solved since Kubo 0.13 (see release notes).

Please upgrade your gateways and clients.
Every Gateway should support Block and CAR responses.

tl;dr summary

When you don't trust gateway, use Block and CAR responses. CAR verification does not require running IPFS node. Clients can leverage standalone tools and libraries such as @ipld/car or ipfs-car:

$ npm i -g ipfs-car
$ curl "https://ipfs.io/ipfs/bafybeiakou6e7hnx4ms2yangplzl6viapsoyo6phlee6bwrg4j2xt37m3q?format=car" | ipfs-car
$ ls ./bafybeiakou6e7hnx4ms2yangplzl6viapsoyo6phlee6bwrg4j2xt37m3q/
1007 - Sustainable - alt.txt
1007 - Sustainable - transcript.txt
1007 - Sustainable.png

Further reading

  • 🎉5
  • 🚀1