Ladder99/ladder99

Update apps to use Nest framework

Opened this issue · 5 comments

tukusejssirs commented

Consider using Nest framework.

This needs to be discussed, as this is a quite big change in how we program l99 and related apps/libs.

bburns commented

@tukusejssirs what is the big picture of Nest?

how would it help organize the code, which mostly run as services interacting with tcp or sql at the moment?

is it for microservice architecture, with interacting rest apis?

tukusejssirs commented

what is the big picture of Nest?

IMO Nx is better for apps/libs organisation. While Nest also supports some simple monorepo structure, Nest is primarily a backend Node.js framework that helps to use SOLID principle (among other things). It also contains some many official and unofficial (let’s say) plugins that eases out configuration and use of some stuff (like Redis, REST, GraphQL, Commander, ORM, OpenAPI/Swagger, Fastify or Express web server, validation hooks, …). @bburns, at the very least, you should go through the sidebar of Nest docs (Overview, Fundamentals, Techniques and Recipes at least), there you can find all the basic stuff what Nest could be used for. Basically, we can build using Nest:

how would it help organize the code, which mostly run as services interacting with tcp or sql at the moment?

At the moment? IMHO it won’t help much now, as it requires rewriting (moving) the code (apps/libs) to Nest (potentially within Nx monorepo). IMHO if you find some spare time, you should try out using Nest yourself and build your first Nest app (probably a web server with some REST API endpoint). You can use Insomnia (prefered by me) or Postman (or other similar app) to test the endpoints (you probably know those tools already 😉).

is it for microservice architecture, with interacting rest apis?

Yes, Nest can be used to built gateway and microservices too, while microservices could communicate with each other programmatically (over various transports, like TCP, gRPC, NATS, Kafka, MQTT, Redis, RMQ; each microservice can be configured communite over a single transport, obviously 😉).

As for REST/GraphQL APIs, those need a web server (Fastify/Express) thus those need to be in a non-microservice app (gateway) which connects all the microservices.

tukusejssirs commented

IMHO the roadmap should be something like this:

  1. Convert all services in the repo to be Nest microservices:
    • we should use MikroORM;
    • consider what transport (TCP, gRPC, MQTT, WS, etc) should be used to communicate with the microservices;
    • in the first step, they should work basically the same way as they do right now, however, later on they should be more or less dumb, i.e. they should not start monitoring unless API gateway (or something else using the transport layer of that microservice) instructs it to start the monitoring (either on l99 startup or when user requests something to do);
  2. Create an API gateway to interact with the microservices and manage the user-facing configuration:
    • start/stop one/many/all microservices;
    • start/stop monitoring of a feature of one/many/all machines;
    • machine management (CRUD operations);
    • machine monitoring management (start/stop monitoring);
    • etc;
  3. IMHO fanuc-driver should be eventually moved into this repository:
    • it will be much easier to keep the l99 and fanuc-driver versions in sync;
    • we could support only a single version for all microservices (l99 services and fanuc-driver);
    • each microservice (be it l99 services or fanuc-driver) could still be used on its own, however, API gateway will make it easier to work with them;
  4. CLI app (currently in BASH scripts) could besides managing the docker images and containers also interact with the API when it is up.

The role of API gateway is to provide an API (GraphQL or REST, or both) to communicate and manage the microservices, even manage some user-facing configuration (like machine management, machine monitoring management, workshift management, etc). The microservices could communicate with each other using the tranport layer (TCP/gRPC/MQTT/WS/etc) or using Redis or whatever else we would need, however, user could interact with them only via the API gateway.

Moreover, I think that we should create a web app (a website if you wish) instead of using Grafana. I have no real experience with Grafana, however, as it is a general purpose tool to visualise data, it has some drawbacks. I don’t say we should not support it, I say we should create a simple alternative frontend (in Svelte or React or whatever other frontend framework you think is the best) at the very least with the following stuff:

  • dashboard with real-time data of all machines (with search, filter, etc);
  • historical data of each and every feature (monitoring scope);
  • auth’n, auth’z, user management, session management (there should be two types of user levels: admins and regular users; possibly we could create a ‘superadmin’ account with a password in some config file as a way to prevent lock-ins).

The web app should be responsive, we could also create a desktop and phone app using the same code to connect to API gateway (user would need to fill in an IP address or we could somehow look in the LAN for it, e.g. using a predefined, publicly available API endpoint like /api/app/version that would contain predefined data).

@MRIIOT, @bburns and others: WDYT? Is this acceptable mid-term plan for l99? Of course, the details need to be ironed out and discussed.

bburns commented

I think that we should create a web app (a website if you wish) instead of using Grafana

Hmm, maybe so, though would be expensive to replace all the grafana functionality.

This is the grafana plugin that allows html inputs, which we could use for simple uis - https://grafana.com/grafana/plugins/volkovlabs-form-panel/ - eg updating setup allowance values per machine.

For the admin stuff, maybe a svelte/react app tho?

tukusejssirs commented

Hmm, maybe so, though would be expensive to replace all the grafana functionality.

I didn’t say it should be done as a top priority nor that there will be feature parity from the first version of the frontend.

As for the expensivity, yes, it’d take some time to implement all feature we currently have in l99, however, we don’t use all the features of Grafana, only a subset of them, thus we need to implement only stuff we actually use.

This is the grafana plugin that allows html inputs, which we could use for simple uis - https://grafana.com/grafana/plugins/volkovlabs-form-panel/ - eg updating setup allowance values per machine.

I haven’t use that plugin, however, it feels like double-processing the data (l99 services → Grafana → Grafana plugin → frontend app). I think we should try to make it simpler: l99 services → l99 API gateway → frontend app. Note that from my point of view, l99 servicess and l99 API gateway are to be collectively called l99 backend, thus the flow could be simplified to backend → frontend.

In general, IMHO everything Grafana does on the backend could be done in Nest app (be it in a microservice or API gateway). Therefore we can do the same job in our side with a configuration if l99 should output the data to Grafana or frontend app or both.

For the admin stuff, maybe a svelte/react app tho?

I think we should have single frontend app for both admin/config stuff and real-time/historical data visualisation, at least that should be our final target. We could reach that final target by smaller steps, thus we can first add admin/config stuff, then add the real-time/historical data.

That said, getting historical data should be fairly simple, once we have an app with Nest + ORM + GQL (so that the frontend app won’t have to make direct queries in the database; we might even want to pre-process the data before returning it via API).

As for the real-time data, we need to get them either via GQL subscriptions (using MQTT as transport, as GQL does not support retaining messages; this way we could provide a single API for both real-time and historical data, as well as configuration stuff) or via MQTT (this way we could have two APIs: MQTT for real-time/current data and GQL for historical data and config).

As for visualisation (other than outputting a simple table), we need to use some chart plugin (e.g. eCharts/echarts-for-svelte](https://www.npmjs.com/package/echarts-for-svelte) or @carbon/charts-svelte/demos) that needs to be configured and fed with the data.