sm(shard manager)

Inspire by

We build sm to handle the backend service which need partition data into several instance(k8s pod or actual machines, use container below), the data in each container may be seen as a job. sm developed base on etcd, to make sure that:

When container changed, some shard moves will be made.
When shard changed, some shard moves will be made.
When shard or container load changed, shard moves(TODO).

Getting Started
- Installing
Concept explanation
- Container
- ShardServer
Example
Question
- Question

Getting Started

Installing

Install etcd, test with etcd-v3.4.13

Install go >=go1.17.1, and run:

go get https://github.com/entertainment-venue/sm/server

go mod tidy

go run main.go --config-file sample.yml

Concept explanation

Container

Container is the resource for shard to run in, and Container establish Session with etcd, the Session is use for the management of key in etcd, management meaning that:

Container goes down, heartbeat(container self or shard in it) all will be recycled.
Only the unchanged properties of app or shard will stay in etcd permanently, make data to be managed as clean as the moonlight.

For more information, you can read the source code for Container.

ShardServer

ShardServer define common protocol which need sharded application who want to integrate with sm:

type ShardInterface interface {
	Add(id string, spec *ShardSpec) error
	Drop(id string) error
	Load(id string) (string, error)
}

You can implement the ShardInterface and inject the implementation into the ShardServer with ShardServerWithShardImplementation, and also wrap common http api to interact with the sm server. The keep http path:

/sm/admin/add-shard
/sm/admin/drop-shard

Please be careful not to use the same path as above in ShardServerWithApiHandler to extend your api.

Example

You can see the test code as tip to understand how to construct you own sharded application:

https://github.com/entertainment-venue/sm/blob/main/server/smserver/testutil_test.go#L5

If you want to no better how the sm server running, see the code below as an entry point:

https://github.com/entertainment-venue/sm/blob/main/server/main.go

Question

Please feel free to report any issue or optimization suggestion in Issues

bearbearkeeper/sm