The Internet of Things (IoT) describes physical objects with sensors, processing ability, software, and other technologies that connect and exchange data with other devices and systems over Internet or any othe communications networks.
IoT Configurator (iotcon) is planned to be a part of the IoT generic data for any system and can be seen as separate IoT system or as IoT subsystem and freely integrated due to the possibility of addition of specific protocols (opcua, modbus, rmq, mqtt, insite and so on), which are implemented through DataSource abstraction, management of required connectors via Indicators as high-level abstraction over sensor/publishers and their variables, orchestrating low-level modules in Transaction-Orchestrator pipe for futher data processing from edge devices. Iotcon ensures that required IoT data lifecycle is processed quickly via connector modules, logically organized through local events and can be operated on at any enterprise level.
Why indicators? - in my opinion, current IoT state is quite complicated to the end-user, it still remains interconnection of unknown terms. With Indicators it's quite evident, that there is some data, which indicates some part of flow (it will relate 1:1 to IoT variable in scope of specific connector). With use of Indicator user is focused only on data, but not connector setup with separate variables. Indicators still maintain the configuration part for sensor and publisher, because connector modules are required to process data. So in fact Indicator flow should be close to Global Variable Devices for basic understanding of the idea
- Monorepo Setup and Management
- Project libraries
- Project board
- API documentation
- Functionality TODO list
Architecture of Iotcon is shown on below diagram, iotcon subsystem can represent a part of Overall IoT System Architecture:
Iotcon dataSources serve as generic source of data standartization, with specific default fields, which will be managed on deeper layers of IoT system. The number of configured fields differ. In current system implementation only basic and not all dataSource fields are taken (they are available for extension), mostly with aim to identify that mandatory data of specific type can be persisted in the system.
Iotcon has the following supported dataSources for connectors (these are added as examples, with the links to documentations):
- INSITE (Intelligent Edge)
- MQTT (MQTT: The Standard for IoT Messaging)
- OPC Unified Architecture (OPCUA)
- RMQ (RMQ for IoT)
- MODBUS (Modbus Refference)
- Wellsite Information Transfer Specification (WITS0)
DataSources management is performed on 3 levels:
- dataSource microservice covers basic CRUD functionality and standard seeding. It is available via gRPC communication
- dataSource module in transactions microservices works with so called
data-source-snapshots.Data-source-snapshotsare cached and supposed to speed up transaction formation, while keeping required fields for transactions and pre-generateddatabusKeyfor the real-time (RT) dataflow (must be unique for each snapshot and indexed). - virtual datasources - implemented via
data-source-snapshotsonly, cover specific cases when no real dataSource exists provided by user and system is supposed to handle it itself (check section for primary modules functionality)
Iotcon uses IoT Orchestrator to perform orchestration for IoT side, which includes the following base functionality (futher development could be required depending on the subsystem integration/setup):
- iot modules schemas and templates management
- transactions processing and logging
- discrete iot modules whole configs storage and retrieval
IoT module should be considered a base functionality item and is itself a connector (sensor/publisher) with specific variables. Manipulations over modules are performed via transactions, each one is supposed to have some number of operations for modules. Operation holds transactionId field, as a mark for transaction service to identify the result of transaction, mode and config fields (each config depends on dataSource type). Before the module creation there is a need to specify schema and template that are also different and depend on dataSource type.
Each transaction must be logged and linked as 1:M relation to operations (because each transaction can have a link multiple performed operations, each of these has own module and mode). Transaction can have 2 generic states (complete and not complete, that are presented by boolean field isComplete). Failed transactions must still have logged moduleIds and operationModes.
Modules storage (precisely their configs) is performed in JSON stringified format as generic approach in current implementation for extensible solution, because types of possible configs can vary in different IoT systems. On orchestration level all operations need to be generic so modules are not differentiated explicitly into sensors or publishers (like on transaction level). Still it is possible to extend to different nested schema types, if needed.
The following limitations also apply:
- All transactions automatically marked as complete and cannot fail (no integration with external IoT/Edge system)
- Nested stored config types absent on the level of Orchestrator
Variables usage at Iotcon is based on the understanding, explained by AWS for IoT Sitewise (AWS IoT Sitewise) and represents an asset of properties. Because Iotcon uses different dataSources (not precisely AWS IoT Sitewise) the differentiation of variables has been provided.
Differentiation of variables for transaction creation is currently based on Data Source, in scope of which this variable is supposed to be used. Primary key (PK) is unique through out all the variables and unique key (UK) is unique in scope of selected connector (check diagram below for interactions):
- Variable name - unique name of variable in scope of connector
- Indicator key - used to link variable to indicator and allows correct data flow (exists in 1:1 relation on the connector module level)
- UoM (Unit of Measurement) - definite magnitude of a quantity, defined and adopted by convention
- UoC (Unit of Class) - classification used for units
The terminology is mostly used to comprehend with understanding of indicators produced inside of the IoT Configurator (internals) and from outside of the system (externals). The aim here stays for possibility of integration with other/external Edge systems. In architectural plane IoT system is not isolated, so creating a basis for externals is logical. The external indicator is also supposed to have unpublished state, which leads to possibility to use it like placeholder for future external data (from MQTT or RMQ brokers). Internals meanwhile are required to have both sensor and publisher (at least default one).
Single API flow includes CRUD for indicators (internals and externals). Standard indicator request body has the following base structure:
{
"id": "uuid",
"name": "string",
"description": "string",
"group": "string",
"tags": ["string"],
"sensor": {},
"publishers": {}[]
}In scope of IoT Configurator sensor and publishers configurations are merged with single variable configuration based on the dataSource to approach to user flow simplification (examples for different configs for implemented dataSources will be provided in docs folder in postman collection).