Implement Ipfs::add
koivunej opened this issue · 0 comments
Well, this was left off #214, #284 and there was no follow-up because it was quite difficult to see what this API would look like before we switched to tokio. Previous toughts at #214 (comment). The best idea I've had for such API would be:
impl Ipfs<_> {
pub async fn add<FileStream>(&self, entries: impl Stream<Item = Result<DirEntry<FileStream>, _>>) -> Stream<Item = Result<AddProgress, _>>;
}
pub enum DirEntry<FileStream: Stream<Item = Result<Bytes, _>> {
Directory(String, ipfs_unixfs::Metadata),
File(String, FileStream),
Symlink(String, String),
}
pub enum AddProgress {
Chunk(String, u64),
// for last element this would be "/" or empty path
Path(String, Cid, u64),
}
Or perhaps we should follow the idea currently experimented way of how ipfs::refs::IpldRefs
works, allowing customization until it has been converted to the stream. At minimum we should be able to control that the add
must process the entries
stream one by one (as is needed in the http API).
DirEntry::Symlink
is String -> String to enforce unixfs utf8 link names, even though they are not enforced at the protobuf level. This would even allow for IpfsPath as the symlink target, which I am not sure if that's ok or not (we definitely do not support them at the moment), nor am I sure if that would be a good idea if it was supported. Strictest setting here would okay the symlink only if it's inside the tree we end up building.
The return value is one aspect to ponder even more: the HTTP api needs to be able to report progress. I don't think there are easy single-task solutions for reporting the progress from just wrapping the MultipartStream
.
I had been pondering on how to add an example of Ipfs::add
in an example/
for #356 before an actual Ipfs::add
is realized but this does seem perhaps even more difficult task than creating Ipfs::add
and then the example.
In the mean time if you stumble upon this issue and need to start adding unixfs files and directory structures please take a look at either:
The http /add
endpoint works on MultipartStream from mpart-async which looks quite a lot like the proposed structure above, but is quite difficult to see/read behind the entangled implementation of /add
. The unixfs/benches/ingest-tar.rs
is more generic but does not save any of the blocks where as the dag test case is more self-contained but has assumptions of how many blocks are created.
EDIT: forgot to mention the HTTP endpoint at first, which works and is passing tests.