/FOSS

Building a Simple Object Storage System (OSS) with F#

Primary LanguageF#

Simple Distributed Object Storage System + NetDisk Backend

Overview

A simple distributed object storage system, and a netdisk backend based on it.

This project is almost implemented in FSharp.

Structure

  • CloudStorage.Common: Common Component
  • CloudStorage.Message: Protobuf-Generated Classes
  • CloudStorage.Server: NetDisk Backend
  • CloudStorage.Storage.ApiServer: Api Server of the OSS
  • CloudStorage.Storage.DataServer: Data Server of the OSS

Usage

  1. Install RabbitMQ, Redis, MySQL

  2. Change configurations in CloudStorage.Common/Config.fs

  3. Start

    # Start Data Servers
LISTEN_ADDRESS=localhost:9991 dotnet ./data/CloudStorage.Storage.DataServer.dll --server.urls "http://0.0.0.0:9991" > data1.out
LISTEN_ADDRESS=localhost:9992 dotnet ./data/CloudStorage.Storage.DataServer.dll --server.urls "http://0.0.0.0:9992" > data2.out
LISTEN_ADDRESS=localhost:9993 dotnet ./data/CloudStorage.Storage.DataServer.dll --server.urls "http://0.0.0.0:9993" > data3.out

# Start Api Servers
dotnet ./api/CloudStorage.Storage.ApiServer.dll --server.urls "http://localhost:8881" > api1.out
dotnet ./api/CloudStorage.Storage.ApiServer.dll --server.urls "http://localhost:8882" > api2.out
dotnet ./api/CloudStorage.Storage.ApiServer.dll --server.urls "http://localhost:8883" > api3.out

# (Optional) Change the OSS location in CloudStorage.Common/Config.fs and recompile
# Start Netdisk backend servers
dotnet ./server/CloudStorage.Server.dll --server.urls "http://0.0.0.0:8000" > serv.out

API

  • GET <oss_data_server>/object/<object_name> - Get object

  • PUT <oss_data_server>/object/<object_name> - Put object

  • GET <oss_api_server>/object/<object_name> - Get object

  • PUT <oss_api_server>/object/<object_name> - Put object

  • GET <oss_api_server>/locate/<object_name> - Locate object

  • GET <netdisk_backend_server>/user/signup - Sign up

  • GET <netdisk_backend_server>/user/signin - Sign in

  • GET <netdisk_backend_server>/user/info - Retrieve my user info

  • GET <netdisk_backend_server>/file/upload - Upload a file

  • GET <netdisk_backend_server>/file/meta - Get file meta

  • GET <netdisk_backend_server>/file/recent - Retrieve recent files

  • GET <netdisk_backend_server>/file/download - Download a file

  • GET <netdisk_backend_server>/file/update - Update a file (currently, just rename)

  • GET <netdisk_backend_server>/file/delete - Delete a file

  • GET <netdisk_backend_server>/file/fastupload - Try fast upload a file

  • GET <netdisk_backend_server>/file/mpupload/init - Init a multi-part upload task

  • GET <netdisk_backend_server>/file/mpupload/uppart - Upload a file part

  • GET <netdisk_backend_server>/file/mpupload/complete - Commit an upload task

  • GET <netdisk_backend_server>/file/mpupload/cancel - Abort an upload task

  • GET <netdisk_backend_server>/file/mpupload/status - Check the status of an upload task