objdeliv

This is a universal temporary large object delivery service.

When some internal artifacts generated by various micro-services need to be sent to the user, you could use objdeliv as a delivery service.

It provides consistent and non-consistent storage drivers, high performance, extremely low memory footprint and very simple and easy-to-implement protocol compared to storing the stuff to Redis or just pipelining them between services.

Installation

First install golang building environment.

Then run:

go build -o objdeliv cmd/main.go

Move objdeliv to $PATH and you can use it now.

If you have just installed, you can use just to build and run, just build to build, just run to run and just build-cross to build for different targets.

Example

Run the following code in your shell and you get a objdeliv server up, running on port 24032, with a LocalDriver which stores all the objects in /tmp/objdeliv as it storage and have a default object expire time of 2 hours (7200 seconds).

go run cmd/main.go -l :24032 -d LocalDriver -o '{"path": "/tmp/objdeliv/"}' -e 7200

Now put an object to it. Example code:

<?php

$sock = fsockopen("127.0.0.1:24032");

$write = fn ($data) => @fwrite($sock, $data);

$write("CONNECT /new-object?expire=60&id=e45d7d04-52b6-473d-bc78-aa40f4b07e73 HTTP/1.1\r\n\r\n");
$write(<<<EOF
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
</head>
<body>
    <h1>My Website</h1>
    <p>This is a test.</p>
</body>
</html>
EOF);

var_dump(json_decode(fgets($sock), true));

fclose($sock);

You will get a result of something like:

array(2) {
  ["status"]=>
  string(7) "success"
  ["id"]=>
  string(36) "e45d7d04-52b6-473d-bc78-aa40f4b07e73"
}

And now navigate to http://localhost:24032/get-object?id=e45d7d04-52b6-473d-bc78-aa40f4b07e73&auto-release=true&content-type=text/html. You will see the object you uploaded!

Now just refresh the page. You set auto-release to true, so the object is now release and cannot be accessed again!

Configure

Three ways can be used to configure objdeliv.

Generally, the priority is: command arguments > environment vars > configure file.

1. Using Environment Variables

Name	Description	Example
`OBJDELIV_LISTEN`	Listen address	`[::]:8080`
`OBJDELIV_DRIVER`	Storage driver name	`LocalDriver` or `MemoryDriver`
`OBJDELIV_DRIVER_OPTIONS`	Storage driver options, in JSON	`{"path": "/tmp/objdeliv/"}`
`OBJDELIV_DEFAULT_EXPIRE`	Default expire time, in seconds	`7200`

2. Using Command Arguments

GLOBAL OPTIONS:
   --config value, -c value, --conf value            Specify the config file path (default: "./configure.yml") [$OBJDELIV_CONFIG]
   --listen value, -l value, --addr value            Listen address [$OBJDELIV_LISTEN]
   --driver value, -d value                          Specify the storage driver [$OBJDELIV_DRIVER]
   --driver-options value, -o value                  Storage driver options, in JSON [$OBJDELIV_DRIVER_OPTIONS]
   --default-expire value, --expire value, -e value  Default object expire time, -1 for unlimited. [$OBJDELIV_DEFAULT_EXPIRE]
   --help, -h                                        show help (default: false)

3. Using Configure File

Use YAML.

An example:

listen: :24032
driver: MemoryDriver
driver-options:
  path: ./dev-temp
default-expire: 7200

4. Storage Driver Options

4.1 MemoryDriver

No options needed.

4.2 LocalDriver

Only one, path, stands for the storage path. If not exist, the program will automatically create it.

Protocol

1. Create an object

This is not standard HTTP!

First client establishes a TCP connection and send a package "CONNECT /new-object HTTP/1.1\r\n\r\n" to the server.

Note: like HTTP, you can send query params in the URL. You can send: id and expire.

If you send id, you need to promise it is a valid UUID string. We will use that instead of generating a new one.

Then the server responses in JSON, with two fields, status and id. status should be success. The id is a UUID string, used to identify the object.

Then you just write your stuff into the socket connection and it works. When finished, just feel free to close the connection.

A sample SDK in PHP:

<?php

class ObjDelivClient
{
    protected $conn;
    protected string $id = '';

    public function __construct(string $addr, ?string $id = null, ?int $expire = null)
    {
        $this->conn = fsockopen($addr);
        $endpoint = '/new-object?';
        if ($id !== null) {
            $endpoint .= 'id=' . urlencode($id);
            if ($expire !== null) $endpoint .= '&';
        }
        if ($expire !== null) {
            $endpoint .= 'expire=' . $expire;
        }
        $pkt = "CONNECT $endpoint HTTP/1.1\r\n\r\n";
        fwrite($this->conn, $pkt);
        $this->id = json_decode(fgets($this->conn), 1)['id'];
    }

    public function getID(): string
    {
        return $this->id;
    }

    public function write(string $bytes): int
    {
        return fwrite($this->conn, $bytes);
    }

    public function __destruct()
    {
        fclose($this->conn);
    }
}

2. Get an Object

Just GET /get-object?id=[THE_ID]&auto-release=[true/false]&content-type=[CONTENT_TYPE]&content-disposition=[CONTENT_DISPOSITION]&pragma=[PRAGMA]&cache-control=[CACHE_CONTROL]&expires=[EXPIRES]

If auto-release is true, after the request the object will be released.

The other parameters are all HTTP response headers that could be customized.

3. Release an Object

Just GET /release-object?id=[THE_ID].

Response is in JSON format.

4. Set Expire Time for an Object

Just GET /set-expire?id=[THE_ID]&expire=[EXPIRE_TIME_IN_SECONDS].