An educational project with an aim to make a fault tolerant distributed file storage. File system supports file reading, writing, creation, deletion, copy, moving and info queries. It also supports certain directory operations - listing, creation, changing and deletion. Files are replicated on multiple storage servers. The data is accessible even if some of the network nodes are offline.
The system is written using django for the name server and flask for storage servers. The following diagram explains the architecture.
All the communication happens via HTTP. The endpoints of each server shown on the diagram above.
Next, let's see example of how the system components interact to upload the file to distributed storage. Here, we assume, that the maximum number of replicas needed in the system is 2.
Let's look closely at each request.
- The client wants to know an ip address of server on which it should upload the file. The client makes a GET request to on /available.
- The name sever returns an ip of the available storage server, in this case, it's the ip of storage2. By available we mean that, the storage is responding status checks.
- The next step is to send the file from client to storage server. The client server sends POST request to storage server with the file it wants upload to distributed storage.
- As soon as the storage server received the file, it sends POST request to name server with file path and file hash to create an entry in the database.
- When the name server created an entry in the database, it checks whether there are enough copies of that file. In our case, the desired number of copies is 2, so the name server chooses another storage to save the file. In the example it chooses storage1 and sends POST request with storage1 ip and the file path to /replicate on the storage2.
- Finally, storage2 sends POST request to /file on the storage1 with the file, which file path it got from /replicate request.
Name server periodically runs checks on storage servers. It sends /status request to each of them and if one of them is not responding, the name server marks it as not available. Let's see what happens if one of the storage servers is not responding and it has a file, wich will have not enough replicas without this storage server. In our case, storage1 is not responding to health checks.
- Name server sends POST /replicate request with file path and ip of the storage1 to the storage server wich has the file needed to be replicated, on the example it's storage2.
- Then storage2 sends the file with the file path it received to storage3.
- After the replication, name server checks whether all files has enough replicas. If not, then the replication process runs again.
That's all, the file has enough replicas now.
Let's see an example of deleting a file, assuming the file is on the storage1 and storage2.
- The client sends DELETE /file request to the name server. The request contains the file path of the file wich is to be deleted.
- The name server finds all the storage servers which has this file and sends DELETE /file request to storage1.
- And the same request to storage2.
There could be a situation when one of the not responding to health checks storage servers became available again. So, this storage server needs to be synced with the system again. Let's see an example of storage1 restoration, assuming it has some outdated file which needs to be deleted.
- The name server sends health check request to storage1.
- Storage1 responds. But it was previously down, so it needs to be synced.
- The name server sends /dump_tree request to storage1 and gets back the files and the hashes which are stored on the storage server.
- The name server compares the file hashes it has in the database and finds out that one of the files is outdated. So, it sends DELETE /file request to storage1 with the file path of the outdated file.
- Before storage restoration there could be a situation when some files did not have enough copies. So, such files have to be found and replicated. The mechanism is the same as replication on file upload.
We have one additional endpoint /restore on the name server that is only accessible for superuser. The request on that endpoint allows to restore the database of the name server by the files on the storage servers.
So, let's get through an example and see how it works. We have a situation when the name server lost its database. The storage servers have to send /register request on the name server to inform it about their ip addresses. So, we have to restart the storage servers. Once the name server knows the ips, superuser can send GET request to /restore and the name server will restore the file tree from servers.
Here is the illustration of restoration process.
- Name server sends GET request /dump_tree to all the storage servers. It gets back file names and their respected hashes.
- Name server chooses the files which have the largest number of copies and their hash is the same.
So, in the tree will be file a.txt with hash 123 and file b.txt with hash 321.
Here is the instruction of how to start the testing setup.
To start the name server run
docker-compose up -d --build
in the name_server directory. To start two storage servers and one client, run
./test_setup.sh
in the project root directory.
One can interact with our system via cli tool. Here are the main commands:
touch [file_name] - creates a file in the current directory
rm [file_name] - deletes file/directory from dfs
mv [src] [dst] - moves a file/directory from source to destination
mkdir [dir_name] - creates a new directory
rmdir [dir_name] - removes an existing directory
ls - lists files in the current directory
cd [dir_name] - changes working directory
push [path] - uploads current state of the file/directory to the dfs
pull [path] - downloads current dfs state to directory
import [host_path] [dfs_path] - copies file/directory from host to dfs folder
export [dfs_path] [host_path] - copies file/directory from dfs folder to host