/hfs

Huang's Chunked Distributed File System

Primary LanguageGoGNU General Public License v3.0GPL-3.0

HFS

Build Status codecov

Huang's Chunked Distributed File System.

My toy.

TODO:

  • concurrently download file

Make it works

  1. install and start etcd:
$ # check https://coreos.com/etcd/docs/latest/dl_build.html
  1. first, create a directory which stores chunks:
$ sudo mkdir -p /hfs/chunks/
$ sudo chown -R <your username>:<your group> /hfs
  1. clone code:
$ git clone https://github.com/jiajunhuang/hfs.git
$ cd hfs
$ make
  1. start chunk server:
$ ./bin/chunkserver
  1. open another terminal, run the client:
$ ./bin/hfsclient upload ~/Downloads/ubuntu-16.04.4-server-amd64.iso
file created, uuid is 60aca0d4-28d9-481b-9a62-460f642664d0
$ ./bin/hfsclient download 60aca0d4-28d9-481b-9a62-460f642664d0
file with UUID 60aca0d4-28d9-481b-9a62-460f642664d0 download successful! origin file name is ubuntu-16.04.4-server-amd64.iso
$ md5sum 60aca0d4-28d9-481b-9a62-460f642664d0
6a7f31eb125a0b2908cf2333d7777c82  60aca0d4-28d9-481b-9a62-460f642664d0
$ md5sum ~/Downloads/ubuntu-16.04.4-server-amd64.iso
6a7f31eb125a0b2908cf2333d7777c82  /Users/neo.huang/Downloads/ubuntu-16.04.4-server-amd64.iso
  1. check chunks:
$ ls /hfs/chunks/
06c520d5-9c01-438f-9760-7004decce303  28749da5-2ad7-4d01-8dff-3b4d5426b0bf  6cba4f9c-4121-4388-b1e9-5015d7aac9bb  d246ea5c-c66e-4ba6-a4aa-b4b3d46f0ef1
0949ed79-4b75-4b9d-aa24-799e725aeffb  39d242e5-4c63-4d0f-8145-629a61e1b4a2  9021960b-b283-4f21-bcbb-9fc425609e99  e97c274b-0467-4295-a488-f917dceb2f70
1dcb04a8-4f3e-4df4-a1bf-28b38de2ac7c  44884d3e-4017-42cd-b0e3-baf3bb0090f2  acb103bb-d140-4cd8-869f-a7ef0f14b1c0
2581cc6c-3ae1-4b8f-8f69-86290d9e2191  4c18bf25-d652-4ed4-ab01-cda48f87a5e6  c3983a62-b770-43df-81c5-c8f2684951ea
  1. check KVs in etcd:
$ ETCDCTL_API=3 etcdctl get "" --prefix=true
/hfs/chunks/06c520d5-9c01-438f-9760-7004decce303
...(ignore the rest)
  1. delete file:
$ ./bin/hfsclient delete 60aca0d4-28d9-481b-9a62-460f642664d0