The repository contains the codebase of the gateway
application which distributes Read and Write operations among the
object storage instances in a cluster.
The demo setup consists of three Minio objects storage instances and a single gateway instance. Find the architecture details in the diagram below.
Follow the instructions below to provision the demo environment locally. Note that the gateway process must have
access to the docker socket /var/run/docker.sock
.
- Docker 23+
- docker-compose
Run to provision the demo setup:
docker-compose up --build
Env Variables Configurations
The gateway process can be configured using the environment variables listed in the table.
Variable Name | Definition | Default |
---|---|---|
STORAGE_INSTANCES_SELECTOR | Selector to identify storage nodes | "amazin-object-storage-node" |
LOG_DEBUG | Logger's debug verbosity level | true |
Given that the demo setup provisioned successfully,
when the script is executed,
then three end-to-end/round-trip upload+download tests are expected to succeed.
The following test files are used:
- The text file with a dummy text sample "foo bar baz";
- The
LICENSE
file from the OpenTofu v1.6.0-alpha1 release; - The
tofu.zip
file from the OpenTofu v1.6.0-alpha1 release. - The
tofu
binary from the OpenTofu v1.6.0-alpha1 release.
Run the command to execute the tests:
make e2etests
Note: the execution requires bash
, curl
, wc
, grep
and diff
.
The gateway can be called from the host machine using the URL: http://localhost:3000. See the API contract in the spec file.
The gateway webserver is a reverse proxy which also functions as the storage cluster's load balancer. Its modus operandi can be described as follows:
- Read and validate HTTP request.
- Return the response with the error message if the request is invalid. Find additional details in the API spec file.
- Read the list of object storage instances available in the cluster using a "service discovery" mechanism.
- Communicate to the storage cluster node:
-
When a read request is received, the gateway "scans" the cluster by sequentially sending the "find command" to each discovered instance over the network. Upon discovery, the "read command" will be sent to the instance and returned data will be proxied to the user. An error message will be returned if no requested data is found, or if the find or read operations fail.
-
When a write request is received, the gateway "scans" the cluster by sequentially sending the "find command" to each discovered instance over the network. Provided data will overwrite existing object upon discovery. If the data is not found, a new object will be created, and the data will be written to the instance selected based on the
objectID
provided by the user. The HTTP status code 201 shall be expected if the write operation succeeds, otherwise an error message will be returned.
---
title: The code diagram of the gateway module architecture
---
classDiagram
class Gateway {
// pkg/gateway/gateway.go
-storageInstancesSelector string
-storageBucket string
-serviceRegistryClient ServiceRegistryScanner
-connectionDetailsReader AuthenticationDetailsReader
-newStorageConnectionFn StorageConnectionFn
+Logger *slog.Logger
+Read(ctx context.Context, id string) io.ReadCloser, bool, error
+Write(ctx context.Context, id string, reader io.Reader, objectSizeBytes int64) error
}
class ServiceRegistryScanner {
// pkg/gateway/gateway.go
<<Interface>>
Scan(ctx context.Context, serviceLabelFilter string) map[string]string, error
}
class AuthenticationDetailsReader {
// pkg/gateway/gateway.go
<<Interface>>
Read(ctx context.Context, instanceID string) string, string, error
}
class ObjectReadWriteFinder {
// pkg/gateway/gateway.go
<<interface>>
Read(ctx context.Context, bucketName, objectName string) io.ReadCloser, bool, error
Write(ctx context.Context, bucketName, objectName string, reader io.Reader, objectSizeBytes int64) error
Find(ctx context.Context, bucketName, objectName string) bool, error
}
class StorageConnectionFn {
// pkg/gateway/gateway.go
<<interface>>
func(endpoint, accessKeyID, secretAccessKey string) ObjectReadWriteFinder, error
}
class Handler {
// internal/restfulhandler/handler.go
+ServeHTTP(w http.ResponseWriter, r *http.Request)
}
class dockerClient {
// internal/docker/docker.go
}
class minioClient {
// internal/minio/minio.go
}
class NewClient {
// internal/minio/minio.go
}
StorageConnectionFn "1" --> "N" ObjectReadWriteFinder
minioClient --|> ObjectReadWriteFinder
dockerClient --|> ServiceRegistryScanner
dockerClient --|> AuthenticationDetailsReader
NewClient --|> StorageConnectionFn
Gateway *-- dockerClient
Gateway *-- NewClient
Handler <|-- Gateway
The gateway module can be extended to use different storage and "service discovery" backends:
- a new service discovery client is required to implement the interface
ServiceRegistryScanner
. - a new secrets manager client is required to implement the interface
AuthenticationDetailsReader
. - a new storage backed client is required to implement the interface
ObjectReadWriteFinder
.
Find a code snippet example below.
package main
import (
"context"
"log"
"log/slog"
"os"
"io"
"strings"
"github.com/kislerdm/object-storage-gateway/pkg/gateway"
)
type myStorageClient struct {
// Attributes of your storage backend's client, i.e. ObjectReadWriteFinder implementation.
}
func (m myStorageClient) Read(ctx context.Context, bucketName, objectName string) (io.ReadCloser, bool, error) {
panic("implement me")
// Definition of the logic to read the object.
}
func (m myStorageClient) Write(ctx context.Context, bucketName, objectName string, reader io.Reader, objectSizeBytes int64) error {
panic("implement me")
// Definition of the logic to write the object.
}
func (m myStorageClient) Find(ctx context.Context, bucketName, objectName string) (bool, error) {
panic("implement me")
// Definition of the logic to find if the object exists in the storage instance.
}
func NewStorageConnection(ipAddress, accessKeyID, secretAccessKey string) (gateway.ObjectReadWriteFinder, error) {
panic("implement me")
// Definition of the logic to initialise your storage backend, i.e. StorageConnectionFn implementation.
return myStorageClient{...}, nil
}
type myServiceRegistryClient struct {
// Attributes of your service registry backend, i.e. ServiceRegistryScanner implementation.
}
func (m myServiceRegistryClient) Scan(ctx context.Context, serviceLabelFilter string) (map[string]string, error) {
panic("implement me")
// Definition of the logic to find storage instances.
}
type mySecretsManagerClient struct {
// Attributes of your secrets manager backend, i.e. AuthenticationDetailsReader implementation.
}
func (m mySecretsManagerClient) Read(ctx context.Context, instanceID string) (string, string, error) {
panic("implement me")
// Definition of the logic to retrieve the auth details required to connected to the storage instance.
}
func main() {
const (
storageInstanceSelector = "my-storage-instance"
storageBucket = "mybucket"
)
gw, err := gateway.New(storageInstanceSelector, storageBucket,
myServiceRegistryClient{}, mySecretsManagerClient{}, NewStorageConnection,
slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{AddSource: true})),
)
if err != nil {
log.Fatalln(err)
}
if err := gw.Write(context.Background(), "foo", strings.NewReader("bar")); err != nil {
log.Fatalln(err)
}
}
Requirements:
- Go 1.21
- Docker and docker compose
- gnuMake/cmake
- Submit the GitHub issue with a bug report or feature request
- Open PR with a code proposal for the code modification, or extension.
Please make sure that the following commands succeed before pushing changes.
- Run unit tests:
make tests
- Run linters:
make lint
The codebase present in the repository is distributed under the MIT license.
The images and graphical material is distributed under the CC BY-NC-SA 4.0 DEED license.