Rivet is both a command-line tool and a Golang package for archiving webpages to IPFS.
Supported Golang version: See .github/workflows/testing.yml
The simplest, cross-platform way is to download from GitHub Releases and place the executable file in your PATH.
From source:
go get -u github.com/wabarc/rivet/cmd/rivet
From GitHub Releases:
sh <(wget https://github.com/wabarc/rivet/raw/main/install.sh -O-)
Using Snapcraft (on GNU/Linux)
sudo snap install rivet
A toolkit makes it easier to archive webpages to IPFS.
Usage:
rivet [options] [url1] ... [urlN]
-host string
IPFS node address (default "localhost")
-m string
Pin mode, supports mode: local, remote, archive (default "remote")
-p string
Pinner sceret or password.
-port int
IPFS node port (default 5001)
-t string
IPFS pinner, supports pinners: infura, pinata, nftstorage, web3storage. (default "infura")
-timeout uint
Timeout for every input URL (default 30)
-u string
Pinner apikey or username.
Stores data on local IPFS node.
rivet -m local https://example.com https://example.org
Stores data to remote pinning services.
rivet https://example.com
Or, specify a pinning service.
rivet -t pinata -k your-apikey -s your-secret https://example.com
Or, stores file locally without any IPFS node.
rivet -m archive https://example.com
package main
import (
"context"
"fmt"
"net/url"
"github.com/wabarc/ipfs-pinner"
"github.com/wabarc/rivet"
"github.com/wabarc/rivet/ipfs"
)
func main() {
opts := []ipfs.PinningOption{
ipfs.Mode(ipfs.Remote),
ipfs.Uses(pinner.Infura),
}
p := ipfs.Options(opts...)
r := &rivet.Shaft{Hold: p}
l := "https://example.com"
input, err := url.Parse(l)
if err != nil {
panic(err)
}
dst, err := r.Wayback(context.TODO(), input)
if err != nil {
panic(err)
}
fmt.Println(dst)
}
If you prefer to disable JavaScript on saving webpage, you could add environmental variables DISABLEJS_URIS
and set the values with the following formats:
export DISABLEJS_URIS=wikipedia.org|eff.org/tags
It will disable JavaScript for domain of the wikipedia.org
and path of the eff.org/tags
if matching it.
Special thanks to @RadhiFadlillah for making obelisk, under which the crawling of the web is based.
We encourage all contributions to this repository! Open an issue! Or open a Pull Request!
This software is released under the terms of the MIT. See the LICENSE file for details.