cachix/devenv

Ensure services are re-usable

DavHau opened this issue · 8 comments

DavHau commented

The whole devenv with services thing is very promising in general.
We already have services defined in nixpkgs and home-manager which are not re-usable in other contexts.
Without any specification on how services must be defined, it's likely that devenv will end up building yet another walled garden ecosystem of services.

Maybe we could create a specification on writing portable service modules.

One idea, could be to ensure that all services are always created via a submodule.
Submodules cannot access the global context of the module evaluation and therefore provide isolation. With this, it won't be possible for a service to set (ore read from) options of arbitrary other services anymore, which should make it more portable
In the example of postgres, the interface could look like this:

options.postgres = {
  type = types.attrsOf types.submoduleWith {
    modules = [./postgres.nix];
  };
  description = "Create instances of postgres";
  default = {};
};

... which then would be used like this:

config.postgres.pg1.enable = true;
config.postgres.pg2.enable = true;

This also solves the problem of only being able to run one instance at a time.
Of course each service needs to get their own data directory etc. That is something that we could define in the spec.

Just pinging @infinisil @roberth . Maybe you have some useful ideas towards that.

thenonameguy commented

Heavy +1 on this.

Now that people are enlightened that Docker might not be the end-all-be-all for local development setups, there is a missing gap in the ecosystem of switching away from it, and it's the missing variable in the
package binary + ? = running service formula.
OCI images usually solve this with entrypoint.sh bash scripts, but these are assuming FHS + root.

NixOS services assume: elevated priviliges + linux kernel + systemd most of the time.
You should not need sudo to start up your local development environment (assuming unprivileged ports are sufficient).
For our Darwin friends, the linux kernel + systemd part is a no-go, so you need virtualization (Docker VM/QEMU = bad perf, networking issues).

I've been using:

Nix Flake provided PATH +
environment variables from https://github.com/numtide/devshell +
https://github.com/F1bonacc1/process-compose service description format + binary +
VCS-checked in "good enough" per-service configuration

to create declarative local services running as normal processes, here are some examples:

nginx-proxy:
  command: "nginx -p $PRJ_ROOT/nginx"
  is_daemon: true
  shutdown:
    command: "nginx -s stop -p $PRJ_ROOT/nginx || true"
    
  zookeeper:
  # serves on (the default ZK) port 2181
  command: "zkServer.sh --config $PRJ_DATA_DIR/zookeeper start-foreground"
  availability:
    restart: on_failure
  environment:
    - "JVMFLAGS=-Djava.net.preferIPv4Stack=true"
    - "ZOO_LOG_DIR=$PRJ_DATA_DIR/zookeeper/logs"
  readiness_probe:
    exec:
      command: "ZOO_LOG_DIR=$PRJ_DATA_DIR/zookeeper/logs zkServer.sh --config $PRJ_DATA_DIR/zookeeper status"
    initial_delay_seconds: 4
    period_seconds: 10
    timeout_seconds: 2
    success_threshold: 1
    failure_threshold: 3

kafka:
  command: "kafka-server-start.sh $PRJ_DATA_DIR/kafka/server.properties"
  availability:
    restart: always
  depends_on:
    zookeeper:
      condition: process_healthy
      
sftp:
  command: >-
    sftpgo portable \
      --directory "$PRJ_DATA_DIR/sftpgo" \
      --sftpd-port $SFTP_PORT \
      --permissions '*' \
      --ssh-commands '*' \
      --public-key "$(<$PRJ_ROOT/local_sftp/ssh_host_rsa_key.pub)" \
      --sftp-key-path $PRJ_ROOT/local_sftp/ssh_host_rsa_key \
      --username sftpgo

# local S3 mock
minio:
  command: "minio server $PRJ_DATA_DIR/minio --console-address localhost:$MINIO_CONSOLE_PORT --address localhost:$MINIO_API_PORT"
  environment:
    - "MINIO_ROOT_USER=minio"
    - "MINIO_ROOT_PASSWORD=justanexampleforgithub"
    - "MINIO_REGION_NAME=eu-west-1"
  availability:
    restart: always
  # https://min.io/docs/minio/linux/operations/monitoring/healthcheck-probe.html
  readiness_probe:
    http_get:
      host: localhost
      scheme: http
      port: $MINIO_API_PORT
      path: "/minio/health/live"
    initial_delay_seconds: 1
    period_seconds: 10
    timeout_seconds: 1
    success_threshold: 1
    failure_threshold: 3

I think the above DSL shows off a few relevant necessities that the solution to this ticket should handle:

  • inter-service dependencies (Kafka depends on ZK)
  • differentiating a healthy/ready process vs a started one (relates to above)
  • daemon processes (nginx by default daemonizes unless told not to: nginx -g 'daemon off;')
  • port assignment (how do you configure port mappings/make them available for the devenv?)
  • the problem of storage of persistent data (persistent data should be opt-out IMO)

While also not highlighting some issues:

  • how to wire up the dependencies of multiple replicas of the same service:
    if I want 2 Kafka clusters running, should they share the ZK cluster? (most probably yes, but how do you allow for both)
  • how to deal with upgrading the data folder of postgres, if you update the major version?
  • how to nuke all persistent data stores (git clean -xf $PRJ_DATA_DIR ?)

Once process-compose gets integrated into devenv
the implementation of this ticket should be simpler. Feel free to use some of the above code to kick-start some service definitions/modules.

  • ❤️1
ibizaman commented

If we're talking about re-using existing tools, I'd definitely explore using disnix. I'm using it on my own server and it works great.

domenkozar commented

I'm currently focused on making language support really good and easy to contribute.

if someone wants to experiment with this, it would be fantastic.

srid commented

I'm forking devenv's services to create a flake-parts version of the same: https://github.com/juspay/services-flake

If services were re-usable, we can avoid that - and share all these modules. services-flake too uses process-compose, so the main thing to decouple here (AFAIU) is devenv's hardcoding of the data directory (config.env.DEVENV_STATE).

  • 👍2
roberth commented

decouple

Could be done by adding an option to the service modules specifically for this value. devenv can assign env.DEVENV_STATE to that, while other consumers of the service modules would assign a value that suits them.

pre-commit-hooks.nix is an example of a set of modules that's consumed by both a devenv and a flake-parts module.
This would be similar, except in this case the service modules are defined by modules that are imported at devenv level rather than in the pre-commit submodule.

The main difference though is that the project is tested in isolation. This of course guarantees that there's no accidental dependency on integration-specific options (e.g. services-flake specific). However, it does assume that contributors test, and not just by applying the patch and then running their own config.

And finally there's the question, should this be limited to process-compose? Some choices:

  1. Reuse nix-processmgmt
  2. Look for lower level (submodule imports) boundaries as suggested by RFC 78
  3. Just roll with nix-processmgmt and keep it simple. (2) can always be factored out later, whereas (1) might be harder to adopt; not sure.
adrian-gierakowski commented

For completeness, link to nix-processmgmt

srid commented

We've implemented multiple services in services-flake, so it is possible do something like this for all services:

{
  config.postgres.pg1.enable = true;
  config.postgres.pg2.enable = true;
}

Feel free to re-use code from our repo as appropriate:

juspay/services-flake@8b8eea9