Ginkgo is a p2sp file transfer tool designed for ML dataset.
go build cmd/ginkgo.go
On server side, just start server with the following command:
ginkgo
On client side, just run ginkgo as a scp command, both directory and file are supported.
ginkgo SrcHost:/path/to/src /path/to/dest
- The
/path/to/src
can be either absolute or relative. - The src and dest path handling behavior is just compatible with GNU scp command
- Client will get other clients list from server side and broadcast itself to them.
- Client chooses the block to download according to consistent hashing result of its serving
host:port
. - Other clients also choose which peer to download the block from by the hash.
- The seed.Seed structure records the blocks and files metadata.
Golang version of gingko
ginkgo
is an alternative form ofgingko
.Gink-go
is the golang versionGingko
.
- HTTP range downloader
- HTTP file server
- Directory support
- Consistent hashing locator
- Client rate limit
- Server rate limit