Scylla Manager

Welcome to Scylla Manager repository! Scylla Manager user docs can be found here.

Scylla Manager consists of tree components:

a server that manages Scylla clusters
an agent running on each node
sctool - a CLI interface to the server

Installing and updating Go

The desired Go version is specified in .go-version file. You can install or update Go version by running make install in go directory. It would install the required version alongside your current version assuming that go is extracted from an official tar package. If you do not have any Go installed at this point you can pass TARGET variable to specify a directory where to install Go.

Procedure:

Run make -C go install TARGET=/path/to/go/sdks/goversion example make install TARGET=~/tools/go/go1.15.5.linux-amd64
Define GOROOT environment variable as <git-root>/go/latest

Installing other packages needed for development

Install Docker
Run installation script ./install-dependencies.sh

Configuring imports formatting in GoLand

If using GoLand update import grouping policy:

Open File -> Settings -> Editor -> Code Style -> Go
Go to Imports Pane
Set "Sorting type" to goimports
Check every checkbox but "Group current project imports"
Press OK

Increase fs.aio-max-nr

In order to run a Scylla it's required to increase the number of async i/o operations limit.

echo "fs.aio-max-nr = 1048576" > /etc/sysctl.d/50-scylla.conf

Running a development environment

Let's start the development environment.

make start-dev-env

This command will:

Build custom Scylla Docker image (testing/scylla)
Compile server, agent and sctool binaries
Start Scylla cluster with 2 DCs 3 nodes each (6 containers)
Start MinIO and Prometheus containers
Start dedicated Scylla container for Scylla Manager datastore

Additionally, you can start container with scylla-manager server.

make run-server

Docker compose environment for test cluster is located in the testing directory.

Once scylla-manager is bootstrapped use ./sctool to add information about the cluster to the manager:

./sctool cluster add --host=192.168.100.11 --auth-token token --name my-cluster

You can ask Scylla Manager to give the status of the cluster:

./sctool status

For other commands consult user manual.

Helpful Makefile targets

Make is self-documenting run make or make help to see available targets with descriptions.

More Makefile targets are available in testing directory.

Running tests

If test environment is running correctly you can run tests with:

make test

This runs both unit and integration tests. You can run them separately with:

make unit-test
make integration-test
make pkg-integration-test PKG=pkg/service/foo

Project testing heavily depends on integration tests, which are slow. Tests should run for couple of minutes. All tests should succeed.

Extending HTTP clients

There are two HTTP APIs that are specified with Swagger format, the API of the Scylla node and the API of the scylla-manager itself.

Client implementations are separated into packages:

scyllaclient which contains client for the Scylla API along with Swagger specification file scylla-api.json and,
managerclient which contains client for the scylla-manager API along with Swagger specification file scylla-manager-api.json.

Both clients are generated automatically by shell scripts. To refresh generated packages from Swagger specification run:

make generate

# Or for generating specific client package ex.
go generate ./managerclient
go generate ./scyllaclient

Sending patches

Develop on dedicated branch rooted at master. Pull master regularly and rebase your work upon master whenever your dev branch is behind. You are allowed and required to force push to your branches to keep the history neat and clean.

Document your work in commit messages. Explain why you made the changes and mention IDs of the affected issues on Github.

Before pushing please run make check and make test to make sure all tests pass.

When satisfied create a pull request onto master. All pull requests have to go trough review process before they are merged.

License

Scylla Manager Source Code is available under Scylla Source Available License (SSAL).

Annamikhlin/scylla-manager