/gateway-conformance

a vendor-agnostic gateway conformance test suite for implementers of IPFS Gateways to ensure compliance with https://specs.ipfs.tech/http-gateways/

Primary LanguageGoOtherNOASSERTION


Boxo logo
Gateway Conformance

A set of GO and HTTP tools for testing implementation compliance with https://specs.ipfs.tech

Official Part of IPFS Project IPFS Specifications ci GitHub release

About

Gateway Conformance test suite is a set of tools for testing implementation compliance with a subset of IPFS Specifications. The test suite is implementation and language-agnostic. Point gateway conformance test at HTTP endpoint and specify which tests should run.

IPFS Shipyard uses it for ensuring specification compliance of the boxo/gateway library included in Kubo, the most popular IPFS implementation, that powers various public gateways, IPFS Desktop, and Brave.

Some scenarios in which you may find this project helpful:

  • You are building an product that relies on in-house IPFS Gateway and want to ensure HTTP interface is implemented correctly
  • You are building an IPFS implementation and want to leverage existing HTTP test fixtures to tell if you are handling edge cases correctly
  • You want to test if a trustless retrieval endpoint supports partial CARs from IPIP-402
  • You want to confirm a commercial service provider implemented content-addressing correctly

Usage

The gateway-conformance can be run as a standalone binary, a Docker image, or a part of Github Action.

Some of the tests require the tested gateway to be able to resolve specific fixtures CIDs or IPNS records.

Two high level commands exist:

  • test (test runner with ability to specify a subset of tests to run)
  • extract-fixtures (allowing for custom provisioning of how test vectors are loaded into tested runtime)

CLI

$ # run subdomain-gateway tests against endpoint at http://localhost:8080 output as JSON
$ gateway-conformance test --gateway-url http://localhost:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 30m

If you are looking for TLDR, see examples.

Docker

Prebuilt image at ghcr.io/ipfs/gateway-conformance can be used for both test and extract-fixtures commands:

$ # extract fixtures to ./fixtures directory
$ docker run -v "${PWD}:/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance:vA.B.C extract-fixtures --output fixtures --merged false

$ # run subdomain-gateway tests against endpoint at http://localhost:8080
$ docker run --network host -v "${PWD}:/workspace" -w "/workspace" ghcr.io/ipfs/gateway-conformance:vA.B.C test --gateway-url http://localhost:8080 --json report.json --specs +subdomain-gateway,-path-gateway -- -timeout 30m

NOTE: replace vA.B.C with a semantic version version you want to test against

Github Action

Common operations are possible via reusable GitHub actions:

To learn how to integrate them in the CI of your project, see real world examples in:

Web Dashboard

Conformance test suite output can be plain text or JSON, which in turn can be represented as a web dashboard which aggregates results from many test runs and renders them on a static website.

The Implementation Dashboard instance at conformance.ipfs.tech is a view that showcases some of well known and complete implementations of IPFS Gateways in the ecosystem.

Learn more at /docs/web-dashboard.md

Commands

See test and extract-fixtures documentation at /docs/commands.md

Examples

Want to test mature specs, while disabling specific specs? Or only test a specific spec (like trustless gateway), while disabling a sub-part of it (only blocks and CARS, no IPNS)? See /docs/examples.md

Releases

The main branch may contain tests for features and IPIPs which are not yet supported by stable releases of IPFS implementations.

Due to this, implementations SHOULD test themselves against a stable release of this test suite instead.

See /releases for the list of available releases.

Development

Want to improve the conformance test suite itself? See documentation at /docs/development.md

Test DSL Syntax

Interested in write a new test case? Test cases are written in Domain Specific Language (DLS) based on Golang. More details at /docs/test-dsl-syntax.md

License

This project is dual-licensed under Apache 2.0 and MIT terms: