HTTP -> Database (Cassandra/Redis/ETC) proxying feature

[conorbros]

Overview

Similar to the websockets -> cassandra functionality in Shotover, this would allow you to communicate with a downstream database with HTTP over Shotover. This would let you connect directly (sort of) to, for example, a Cassandra database from an environment where only HTTP is available.

Use

In Shotover this might be available to users with the following topology.yaml:

---
sources:
  - HTTP:
      name: "http-cassandra"
      listen_addr: "127.0.0.1:6379"
      chain:
        - MessageConvert
              type: Cassandra

        - CassandraSinkCluster:
            first_contact_points: ["172.16.1.2:9044", "172.16.1.3:9044"]
            local_shotover_host_id: "2dd022d6-2937-4754-89d6-02d2933a8f7a"
            shotover_nodes:
              - address: "127.0.0.1:9042"
                data_center: "datacenter1"
                rack: "rack1"
                host_id: "2dd022d6-2937-4754-89d6-02d2933a8f7a"
            connect_timeout_ms: 3000

Implementation
To add this feature to Shotover two pieces would need to be implemented:

• HTTP source. Most of this functionality is already implemented in Shotover for the OpenSearch work.
• a new transform to convert messages between types. Alternatively this functionality could added to the codecs, where they would convert messages of a different type to their output automatically, but I think the transform is more explicit and lets you use. This will just pull the raw bytes of the protocol message out of the HTTP message body and convert into a Cassandra message type within Shotover.