Standardize Storage AskProtocol v2
jennijuju opened this issue · 5 comments
Originally posted by @brendalee, edited by @jennijuju December 13, 2021
Starting this issue to get additional feedback from the community on proposed updates to the storage AsksProtocol v1.1.0. Currently, the ask protocol does not give the client / SP enough information on the likelihood of a storage deal being accepted. Note that there have already been a handful of discussions around storage ask protocol changes with the community (notably in Filecoin Slack #arg channel and other ad-hoc discussions with storage providers).
The goal of AskProtocol v2 is to help with awareness between the client and the storage provider before a deal proposal. Before a client attempts to make a deal, they can send details of their proposed deal, and the storage provider can respond to the proposal (with accept or reject and other details such as price). Note that the ask protocol is not designed to handle back and forth deal negotiations.
Proposed Storage AskProtocol v2 Standard
This proposal is to request change to the current ask protocol and update the version of AskProtocolID
to /fil/storage/ask/1.1.0
, and
Storage AskProtocol v2 Sketch
We need to design a new storage ask protocol. Let's brainstorm what it might look like here.
I think we should keep it very simple still, with a client requesting, and the storage provider responding.
To get started, i'm just going to define a request
and a response
type, and then we can assume thats the protocol.
Request
type StorageAskRequest struct {
ClientAddress Address
PeerId *abi.PeerID //nullable
PieceSize abi.PaddedPieceSize
Duration int64
//Deal start epoch
StartEpoch abi.ChainEpoch
Verified bool
TransferProtocol string
Message string
Signature: The request should be signed by the client.
Open question:
- Should we have `fast retrieval for the request, given that FVM may enable retrievalbility reaasurance?
- Do we need a
Message
field as string as a free-form field set by the client to the storage provider? - Do we need the client to provide the payload CID?
Response
The response should be a signed object containing the request, for context
type StorageAskResponse struct {
Query StorageAskRequest
// WillAccept denotes whether or not the storage provider will accept the proposed deal
WillAccept bool
// RejectionCode should projects the reection reason
RejectionCode uint
// Price per GB per epoch. This is a single price field, verified vs unverified is specified by the Query
Price abi.TokenAmount
// Expiry denotes how long this offer is valid for
// This AskResponse is not binding in any way, this is just for the clients convenience.
Expiry time.Time
//DealStartEpoch is how soon miner offers to get the deal fully onchain (provecommited)
DealStartEpoch abi.ChainEpoch
// Message is a free-form field set by the storage provider to the client
Message string
}
Open Questions:
RejectionCode
: Should we keep the code simple and have a predefined list of error codes, or should it to be a string for customized rejection reason in here. Or should it be a structured field? or left open for people to do whatever they like?- Miner response can contain an epoch that they will try to get the published deal on chain
Initial FIP draft created at #224
Simple Summary
The goal of AskProtocol v2 is to help with the discovery between the client and storage providers before making a storage deal proposal. As the first step of the SP discovery, clients can send details of their proposed deal to the storage provider, and the storage provider should be able to respond to the proposal (with accept or reject, price and additional service offer). Note that the ask protocol is not designed to handle back and forth deal negotiations.
Abstract
AskProtocol v2 aims to improve the existing AskProtocol to help the client understand whether or not a storage provider is likely to accept their deal before they send a signed proposal.
The proposed flow is simple. The client sends an ask request with information about the storage deal they would like to make. The ask response by the storage provider indicates whether or not the deal will be accepted with specifics on conditions (for example, how long the offer is valid for, and what the price to store is). More details about the AskProtocol v2 request/response can be found in the specification section.
Change Motivation
Clients need to be able to share details of the storage deal they want to make. Furthermore, they need to know what capabilities and features a Storage Provider supports, so as to understand what Storage Providers they can successfully make a deal with.
Specification
Client Request of AskProtocol v2
For the AskProtocol v2 request, the client provides the following information:
type StorageAskRequest struct {
ClientAddress Address
PeerId *abi.PeerID
PieceSize abi.PaddedPieceSize
Duration int64
StartEpoch abi.ChainEpoch
Verified bool
TransferProtocol string
Message string
where:
ClientAddress
: is the wallet/account address that the client will be used for signing the deal proposal and making/paying the storage dealPeerId
: is the peerID of the client's node, this is an optional fieldPieceSize
: is the padded size of the piece in the dealDuration
: is the deal durationStartEpoch
: is the start epoch of the dealVerified
: if the deal is a verified dealTransfer protocol
: is the protocol that will be used for data transfer, for example, graphsync, physical. Aside from a list of defined the protocols , the client/storage provider may have the option to specify/add additional transfer protocolsMessage
: is a free-form field set by the client to the storage provider
Signature: the request should be signed by the client address
Storage Provider Response of AskProtocol v2
For the response, the provider will respond with the following information:
type StorageAskResponse struct {
Query StorageAskRequest
WillAccept bool
RejectionCode uint
Price abi.TokenAmount
Expiry time.Time
DealStartEpoch abi.ChainEpoch
Message string
}
where:
Query
: is theStorageAskRequest
that was sent by the clientWillAccept
: is whether the storage provider will accept or reject the dealRejectionCode
: is codes for rejection reasons- Need clearly defined rejection codes
- Start with HTTP codes and build from there
- Option for storage provider to specify additional rejection codes
Price
: is the offering price per GB per epoch.Expiry
: is how long the offer is valid for, this is not a binding information but just for client's informationDealStartEpoch
: is how the storage provider is offering to the the deal fully committed to the network with in a sector- `Message: is a free-form field set by the storage provider to the client
Design Rationale
TBD
Backwards Compatibility
TBD
Test Cases
TBD
Security Considerations
TBD
Incentive Considerations
N/A
Product Considerations
One of the main experiences we want to improve is storage dealmaking and overall Filecoin storage reliability. AskProtocol v2 improvements aim to help with the following:
- Better information on Storage Provider capabilities before client attempts to make a storage deal
- Easier to filter which Storage Providers will accept a client’s storage deals
- Increase storage deal acceptance and success rate
Implementation
TBD
Copyright
Copyright and related rights waived via CC0.
Some discussion points -
Ask request
- Flagging that fast retrieval has been removed, any concerns with this?
- PeerID - part of the libp2p protocol, SP should already have this information (is this redundant?)
Ask response
- DealStartEpoch - this field is for the SP to share when the deal start epoch could be, and should also be able to encompass if the SP wants the client to come back at a later time. How would the SP calculate this (we should have a standard that we push to SPs so it’s not interpreted differently)? - @whyrusleeping thoughts on this?
- RejectionCode - @dirkmc feedback
HTTP doesn’t map very well, would be good for us to delinate a list of error codes and use those.
Currently this is a number, would it be better if this is a string - Thoughts about including option for SP to include client collateral requirement (https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/market/deal.go#L23)
In venus-market, we have gathered the needs of many miners and miners. access the venus-market address directly during the ask process (market url obtained by accessing a fixed contract/config in the future), and return a series of miners and the miner’s demand, and then let the client choose a miner to issue an order.
I would like to put @hunjixin 's thought into more general.
In Filecoin ecosystem, the storage provider just part of the network infrastructure, above that, there could be higher layer services, e.g. deal broker, gateway or proxy, to help on storing data to Filecoin and retrieval from Filecoin network too. In addition, when we consider retrieval, there could be services particularly for retrieval too, that means, the fast retrieval could be done by other service nodes.
And all these services bridging between the clients and storage providers could be possibly provided by smart contracts when FVM is available.
In the v2 design, should we make it more flexible to not limit to storage provide nodes and end clients, or we design a delegate or trust protocol for the market in another FIP, while having this protocol support it?
I would love to evaluate this proposal, but can't do much without much more description of the perceived problems, motivation for specific changes, and design rationale.