[LocalNet] Run LocalNet development set up with operator
Closed this issue · 8 comments
Objective
We want to run developer environments utilizing this operator. Current implementation relies on docker-compose: https://github.com/pokt-network/pocket/tree/main/docs/development#running-localnet
The idea is that if we utilize this operator to not only maintain workloads on production Kubernetes clusters for dev, test and main-nets, but also for local-nets, we will have this product polished well for 3rd party use along with V1 client.
Goals
- Functionality should be (at least) on par with docker-compose set up, meaning dependencies are shipped and working well with network actors:
- metrics collection
- logs collection
- grafana dashboards
- make targets that currently interact with docker compose should work agains set up inside Kubernetes
Deliverable
- LocalNet runs on Kubernetes, workloads are controlled by the operator
- Metrics and Log targets are discovered automatically, Grafana can consume the output
- Parity is reached by following the instructions in the Development Guide to achieve the exact same flow & outcome
Non-goals / Non-deliverables
- The current requirement is to only run 4 validators, though in future that number will have to be dynamic
General issue deliverables
- Update the appropriate CHANGELOG
- Update any relevant READMEs (local and/or global)
- Update any relevant global documentation & references
Creator: @okdas
Co-Owners: TBD
Did some research on how to approach this. InternalSet resource that will manage other CRDs would be ideal, taking a look if can implement this quickly or (for now) stick to utilizing Validators directly.
@okdas I believe another deliverable should be:
- Parity is reached by following the instructions in the Development Guide to achieve the exact same flow & outcome
Wdyt?
Opting-in to use https://tilt.dev/ because this way we are able to integrate Kubernetes operator and client – both running with hot reloaded code from local directories.
Opting-in to use tilt.dev because this way we are able to integrate Kubernetes operator and client – both running with hot reloaded code from local directories.
I read what they have in store and support this 👍👍
We've made a decision to first run LocalNet without operator, but utilizing tilt to control resources on local Kubernetes directly, as reflected in the PR pokt-network/pocket#354.
We are going to circle back on this to also support running pocket nodes with hot reloading from pocket codebase because there is a need in that for operator development (and, eventually, dogfooding the operator itself!). This set up did not work as intended and caused instabilities and disruptions of development process, but I believe it can be addressed.
We've made a decision to first run LocalNet without operator, but utilizing tilt to control resources on local Kubernetes directly, as reflected in the PR pokt-network/pocket#354.
Could you provide more details on why this is and so it's easier to pick up in the future (for yourself or others).
Could you provide more details on why this is and so it's easier to pick up in the future (for yourself or others).
The process involved too many moving pieces and not all pieces are stable enough, it was hard to sync/align between them:
- There were two Tiltfiles - one for pocket and a second one for pocket-operator
- pocket-operator is provisioned with operator-builder which is a codegen solution but is an additional step in this set up
- I initially relied on postgres-operator to provision the databases, which has it's own opinions and is great for production usage, but not to provision databases locally (too many requirements to be met)
- config management between pocket-operator and pocket codebase is not easy, but this will be changed if we continue to use operator - we can import pocket configuration structs because both projects are written in golang
In the end, all issues are resolvable, but I wanted to deliver LocalNet sooner, and it was easier to skip operator. We can revisit this once operator is more mature!