Core Module Inbound Connectors

[dixonwhitmire]

The LinuxForHealth HealthOS uses connectors to receive and transmit data. This epic supports the development of inbound connectors used to acquire data. Connectors in scope for this epic include:

• KafkaConsumer (uses aiokafka dependency)
• NatsClient (uses nats.py dependency)
• /ingress RestEndpoint (uses fastapi dependency)

The KafkaConsumer and NatsClient "pull" data from external sources while the /ingress endpoint allows external clients to "push" data to the HealthOS.

These connectors are configured as part of the Core Service Definition

Each of these connectors utilize asyncio based libraries to integrate with their respective data sources. As a result we will need to provide an "event loop" to the HealthOS so that these libraries can execute concurrently.

To keep things simple in this first iteration, we will create a Fast API application as a default component that is spun up during "startup". The app will support an "admin" api for interacting with connectors (implemented as asyncio tasks), and an optional "/ingress" connector endpoint. This allows us to use the event loop that Fast API provides. Due to this detail, the Fast API issues will be the first issues worked as it is provides the execution environment for the other connectors.

The flow for data ingress uses Nats subjects and subscribers for validation:

• connector receives data
• connector publishes data to lfh.healthos.ingress
• Nats client subscribed to lfh.healthos.ingress validates the data
• Data validation errors are sent to lfh.healthos.ingress.errors

Success criteria (scoped to development environment):

• The development environment includes containers to support "external services"
• The development environment includes a startup process to launch the core service using the core service configuration.
• The development environment includes tooling to transmit HL7, FHIR, X12, and dicom payloads to the inbound connectors.
• The development environment includes a logging facility to monitor the movement of data throughout the system
• Validation errors are transmitted to lfh.healthos.ingress.errors when appropriate