🏍️ DeStor REST API for Filecoin
Accelerating data onto Filecoin!
Table of Contents
Background
The DeStor REST API for Filecoin is an interface to easily propel data onto the Filecoin network. The REST API is implemented here in a service named 'motion'. This service aims to create an easy path for independent software vendors to integrate Filecoin as a storage layer.
Install and setup
Prerequisites
-
A Filecoin wallet to make deals with, for which you are in possession of the private key. Various options for obtaining a wallet can be found here (https://docs.filecoin.io/basics/assets/wallets/).
-
A server (bare metal or VM) to run Motion on, with Docker Engine installed on that server with the Docker Compose plugin included. Recommended hardware requirements for servers are:
- Because we run docker, Linux variants are preferred for the OS
- Recommend at least >500GB disk space available for staging data. The complete Filecoin deal making process takes up to 3 days, and you will need to hold all data until deal making is complete. So the amount of free space you will need is roughly the amount of data you want to onboard per day times 3.
- In general, we do not believe Motion is processor or memory intensive, but a machine with at least 32GB of RAM is optimal
- You will also need
git
installed on this machine. - The server must have a static IP and/or a domain name that points to it, with a port open so you can transfer data from Motion to Filecoin storage providers
Setting up motion for the first time
Start by cloning this repository:
git clone https://github.com/filecoin-project/motion.git
Before you can run motion, you must configure it. First, from the motion directory, copy .env.example
to .env
:
cp .env.example .env
Now open .env
and set the required values for your instance of motion. At minimum, you need to set the following values (excerpt from .env.example
file):
# Comma seperated list of storage providers motion should make storage deals with
# You must set this value to contain at least one storage provider for motion to
# work
MOTION_STORAGE_PROVIDERS=
# The private key of the wallet you will use with motion, in hexadecimal format.
# This is the output of `lotus wallet export ~address~` if you are using lotus
# If you are obtaining a wallet through another method follow your wallet providers
# instructions to get your wallet's provider key
MOTION_WALLET_KEY=
# This is the domain/IP you will expose publicly to transfer data to storage providers
# When you initialize the singularity docker setup, it will start a server to deliver
# content to storage providers on localhost:7778. However, you will need to either
# open this specific port on your firewall, and set this value to http://~your-static-ip~:7778
# or you will need to setup reverse proxying from a dedicated web server like NGinx
SINGULARITY_CONTENT_PROVIDER_DOMAIN=
Again, you could open an issue if you need assistance setting up your wallet, your server to expose a port for data transfers publicly on your server.
As needed, you can also set additional values in your motion `.env`` file for more custom configurations.
Usage
Once you've configured your .env
, you're ready to start motion. From the local repository, simply run
docker compose up
(if you don't want to see log messages directly in your terminal, run docker compose up -d
)
You'll know your motion is up an running when you see a message like this in the docker logs:
2023-09-07 17:49:57 motion-motion-1 | 2023-09-08T00:49:57.530Z INFO motion/api/server server/server.go:53 HTTP server started successfully. {"address": "[::]:40080"}
Your copy of motion is now running. The Motion HTTP API is now running on port 40080 of your server's localhost.
Store blobs
To store an example blob, use the following curl
command :
echo "fish" | curl -X POST -H "Content-Type: application/octet-stream" -d @- http://localhost:40080/v0/blob
The response should include a blob ID which you can then use the fetch the blob back. Example:
{"id":"ad7ef987-a932-495c-aa0c-7ffcabeda45f"}
Storing onto Filecoin
Motion will begin saving data to Filecoin when it's holding at least 16GB of data that hasn't been backed up with a storage provider.
If you want to test storing an actual filecoin deal, the following simple script will put about 20GB of random data into motion:
for i in {0..20}; do; head -c 1000000000 /dev/urandom | curl -X POST --data-binary @- -H "Content-Type: application/octet-stream" http://localhost:40080/v0/blob; done
This should be enough to trigger at least 1 Filecoin deal being made from Motion
Retrieve a stored blob
To retrieve a stored blob, send a GET
request to the Motion API with the desired blob ID.
The following command retrieves the blob stored earlier:
curl http://localhost:40080/v0/blob/ad7ef987-a932-495c-aa0c-7ffcabeda45f
This should print the content of the blob on the terminal:
fish
Alternatively, you can browse the same URL in a web browser, which should prompt you to download the binary file.
Check the status of an uploaded blob
In addition to retrieving data for a blob, you can also check the status of its storage on Filecoin:
curl http://localhost:40080/v0/blob/ad7ef987-a932-495c-aa0c-7ffcabeda45f/status | jq .
(jq
being used to pretty print here -- make sure it's installed on your machine. you don't need to pipe to jq but the output will be more readable)
{
"id": "ad7ef987-a932-495c-aa0c-7ffcabeda45f",
"Replicas": [
{
"provider": "f1234",
"status": "active",
"lastVerified": "2020-12-01T22:48:00Z",
"expiration": "2021-08-18T22:48:00Z"
}
]
}
API Specification
See the Motion OpenAPI specification.
Status
🚧 This project is currently under active development.
Local Development
To set up filecoin-ffi
dependencies, run:
make build
This is only necessary to run once. After that you can use the regular go build
command to build Motion from source.