upcast: A Haskell repository from 4z3

Upcast is a declarative cloud infrastructure orchestration tool that leverages Nix. Its nix codebase (and, by extension, its interface) was started off by copying files from nixops.

Quick start

upcast - infrastructure orchestratrion

Usage: <interactive> COMMAND

Available commands:
  run                      evaluate infrastructure, run builds and deploy
  infra                    evaluate infrastructure and output ssh_config(5)
  infra-tree               dump infrastructure tree in json format
  infra-debug              evaluate infrastructure in debug mode
  instantiate              nix-instantiate all NixOS closures
  build                    nix-build all NixOS closures
  build-remote             nix-build all NixOS closures remotely
  nix-path                 print effective path to upcast nix expressions
  install                  install nix environment-like closure over ssh

## see https://github.com/zalora/nixpkgs
$ export NIX_PATH=nixpkgs=/path/to/zalora/nixpkgs:$NIX_PATH

## prepare credentials
$ awk 'NR==1 {print "default", $1, $2}' ~/.ec2-keys > ~/.aws-keys # assuming you used nixops

## build upcast
$ cabal install

## fill in your ec2 vpc account information (look into other examples/ files to provision a VPC)
$ cp examples/ec2-info.nix{.example,}
$ vim examples/ec2-info.nix

## execute the deployment
$ upcast run examples/vpc-nix-instance.nix

For more examples see nix-adhoc.

Goals

simplicity, extensibility;
shared state stored as nix expressions next to machines expressions;
first-class AWS support (including AWS features nixops doesn't have);
pleasant user experience and network performance (see below);
support for running day-to-day operations on infrastructure, services and machines.

Notable differences from NixOps

Expression files

You can no longer specify the machine environment using deployment.targetEnv, now you need to explicitly include a module instead (currently available: <upcast/env-ec2.nix>).
You can deploy an EC2 instance that does not use nix in its base AMI by using deployment.nix = false; (you won't be able to deploy a nix closure to such machine)>

Operation modes

The only supported command is run (so far). No create, modify, clone, set-args, send-keys.
NixOps SQLite state files are abandoned, separate text files (json dict for state and a private key file) are used instead;
Physical specs are removed
- Identical machines get identical machine closures, they are no longer parametric by things like hostnames (these are configured at runtime).

Infrastructure services

(this is what used to be called resources in NixOps)

New: EC2-VPC support, ELB support;
Additionally planned: AWS autoscaling, EBS snapshotting;
Different in EC2: CreateKeyPair (autogenerated private keys by amazon) is not supported, ImportKeyPair is used instead;
Not supported: sqs, s3, elastic ips, ssh tunnels, adhoc nixos deployments, deployments to expressions that span multiple AWS regions;
Most likely will not be supported: hetzner, auto-luks, auto-raid0, /run/keys support, static route53 support (like nixops);

Motivation

Cookbook

Nix-profile uploads

Achieving productivity

tl;dr: do all of these steps if you're using a Mac and/or like visiting Starbucks

Remote builds and remote deployments

Unlike Nix distributed builds packages are not copied back and forth between the instance and your local machine.

upcast build-remote -t hydra.com examples/vpc-nix-instance.nix

The former is roughly equivalent to:

builder=user@hydra.com
upcast instantiate examples/vpc-nix-instance.nix | {
  read drv;
  nix-copy-closure --to $builder $drv 2>/dev/null &&
    ssh $builder "nix-store --realise $drv 2>/dev/null && cat $(nix-store -qu $drv)"
}

This outputs a json string that looks like {"node":"/nix/store/72q9sd9an61h0h1pa4ydz7qa1cdpf0mj-nixos-14.10pre-git"}, you can use it as a part of your deployment without having to upload packages from your local store using run -f:

upcast run -m '{"node":"/nix/store/72q9sd9an61h0h1pa4ydz7qa1cdpf0mj-nixos-14.10pre-git"}' -f user@hydra.com examples/vpc-nix-instance.nix

If you want to update your existing systems as part of your CI workflow without having to talk to infrastructure services, you can cook something like this:

upcast infra examples/vpc-nix-instance.nix > ssh_config

awk '/^Host/{print $2}' ssh_config | \
    xargs -I% -P4 -n1 -t ssh -F ssh_config % nix-collect-garbage -d

awk '/^Host/{print $2}' ssh_config | \
    xargs -I% -P4 -n1 -t upcast install -t % $(upcast build-remote -A some-system blah.nix)

Making instances download packages from a different host over ssh (a closure cache)

If you still want to (or have to) build most of the packages locally, this is useful if one of your cache systems is accessible over ssh and has better latency to the instance than the machine you run Upcast on.

The key to that host must be already available in your ssh-agent. Inherently, you should also propagate ssh keys of your instances to that ssh-agent in this case.

export UPCAST_SSH_CLOSURE_CACHE=nix-ssh@hydra.com

SSH shared connections

ControlMaster helps speed up subsequent ssh sessions by reusing a single TCP connection. See ssh_config(5).

% cat ~/.ssh/config
Host *
    ControlPath ~/.ssh/master-%r@%h:%p
    ControlMaster auto
    ControlPersist yes

Known issues

you have to use zalora's fork of nixpkgs with upcast
state files are not garbage collected, have to be often cleaned up manually;
altering infra state is not supported properly (you need to remove using aws cli, cleanup the state file and try again);
word "aterm" is naming a completely different thing;
i hardcoded x86_64-linux in some places 👼

Note: the app is currently in HEAVY development (and is already being used to power production cloud instances) so interfaces may break without notice.

More stuff

The AWS client code now lives in its own library: zalora/aws-ec2.

4z3/upcast