/pm4ml-k3s-iac

iac for k3s cluster to install pm4ml

Primary LanguageJinjaApache License 2.0Apache-2.0

Mojaloop K3S Bootstrap

A minimal PoC of mojaloop running on k3s

Includes

  • ingress controller (nginx-ingress, ambassador or traefik)
  • external-dns (Automatic DNS registration in the configured domain)
  • cert-manager (Automatic SSL Cert creation using LetsEncrypt)
  • longhorn (Distrbuted storage driver)
  • gitlab (CI/CD within the deployed system)
  • wireguard (for VPN)
  • mojaloop (Deployed with default mojaloop helm chart)

Instructions for use

Prerequisites

  • awscli must be installed and configured for creating infrastructure and terraform backend. This is available in the package repos of most Linux distributions. Ensure you're using v2:
    $ aws --version
    aws-cli/2.1.17 # blah blah
  • Ensure you have AWS access keys (for IAM users) configured in your environment with:
    $ aws configure
  • GNU Make 4.x is required. By default, MacOS ships with Make 3.x, see (https://stackoverflow.com/questions/43175529/updating-make-version-4-1-on-mac) for details. Check with
    make --version
  • Terraform >= 0.13.5. Find this in your package manager, or use make install-terraform to install in this directory.
  • Ansible

Bootstrap a new environment

  • Clone this repo

    git clone https://github.com/modusintegration/mojaloop-k3s-bootstrap
  • Perform initial configuration;

    make config

    Enter any required values, or accept the given defaults:

    make config

  • Create the terraform backend;

    make backend
  • Initialise terraform (this will also install the required version of terraform and the ansible terraform provider if needed)

    make init
  • Create the infrastructure;

    make apply
  • Install k3s, ingress controller, external-dns and cert-manager

    make k3s
  • Destroy complete infrastructure

    First, disconnect VPN

    find wireguard.clients -name '*.conf' -exec sudo wg-quick down \{\} \;

    Then

    make destroy
  • Above shows a basic lifecycle only, see a full list of targets available for installation by running

    make help

Existing environment

  • To run commands against an existing bootstrapped environment;
    • Checkout the repo
      • Note: If using WSL on windows, make sure to clone inside the WSL Shell so that permissions are preserved
      git clone https://<your gitlab url>/IaC/k3s-bootstrap.git [preferred repo directory]
    • Initialise terraform
      make init
    • Done!

kubectl access

  • To connect to the master and run kubectl/helm commands; You'll require the ansible-inventory utility, which is likely bundled with ansible for your OS/distribution.

  • To retrieve a kubeconfig file for local access to the cluster:

    Get VPN configs:

    make vpn

    Bring up VPN:

    sudo wg-quick up $PWD/wireguard.clients/client1.conf

    Get the kubeconfig file (could take up to a minute):

    make kubeconfig

    Access cluster:

    export KUBECONFIG=$PWD/kubeconfig
    kubectl get pods

Bastion host

The bastion host is used as an SSH jump host only, there are no kubectl/helm cli tools available.

Ansible playbooks

  • Generally, all installation should be made available as make targets as per above, however if you need to deploy changes via ansible, or re-run a specific playbook, you can do so;
    • From within above initialised repo, run;

      make ansible-playbook -- <playbookname>.yml

      Note: Further additional arguments to ansible can be passed after the playbook name, e.g. tags or verbose flags for debugging

      make ansible-playbook -- k3s-infra.yml -t cert-manager -vvv
    • If you execute make ansible-playbook without additional arguments, a list of available playbooks will be displayed

Monitoring Stack - EFK vs Loki

See README-monitoring.md

OnPrem specific instructions

See README-onprem.md

Vault integration for payment manager

See README-vault.md

TODO

  • Move terraform module out to a seperate repo as a module which can be referenced
  • Integrate keycloak for kubernetes RBAC