/omg-acurite-filter

Primary LanguageTypeScript

Background

Requirements

  • Gather in MQTT messages for specific types of messages.
  • Save the last X minutes of data.
  • Compare the most recent data to the last X minutes of data. If the data seems 'unusual' discard the message.
    • Define "unusual"
      • Simple:
        • Temperature is + / - 2.5 C of the last received temperature
        • Humidity is + / - 5% of the last received temperature.
        • Each time we receive a reading, we discard any values more than X minutes old.
        • If we have no saved readings, then the reading is automatically valid.
        • This means that if...
          • We don't typically receive a reading, it will automatically be considered valid. Mixed bag here. This also means that weak signals (the most likely to be corrupt) will be more likely to be forwarded
          • If we receive a reading that seems wacky (but is actually valid) it will auto-correct after 5 minutes.
          • If we log a wacky value as the 'valid' reading, it will auto-correct after 5 minutes.
  • Every Y seconds, send the 'latest' valid data reading
    • Only send new readings though, if we already sent it, flag it as such.

Enhancement ideas

  • Monitor memory usage. If it goes beyond X, auto-restart the service
  • Save stats:
    • Invalid Readings Stats
      • Prev reading - Invalid Reading
      • Date//Time of bad reading
      • Age out individual readings?
      • Total # per device per receiver
  • REST API?
    • Stats?
  • Publish statistic information to Home Assistant via MQTT & MQTT self discovery.
  • Reverse route MQTT messages from the forwarded destination back to the original sources.
  • Forward OMG device information & reverse route MQTT controll messages back to the OMG devices.
  • Setup PM2 service on VM

Other validation ideas

  • Many times we receive multiple copies of the same message, each from a different receiver.
    • The messages arrive nearly simultaneously.
      • Maybe we store any messages received in the last X ms into a special buffer?
      • We could look back in the buffer for messages received in the last X milliseconds.
      • When sending messages via our scheduler, we could ignore very recently received messages to allow msg validation to fully run, but this would complicate the send logic.
    • All messages in the buffer should be equal.
      • If we have 3+ messages, we can throw out the odd one.
      • What do we do when we have 2 messages, and one doesn't agree?
      • [] = All message buffer
      • () = Within last 100 ms
      • {t-Xs} = Message received X seconds ago
      • [ 66 {t-15s}, 66 {t-15s}, 66.1 {t-10s}, (63.6, 66.1) ]
        • This looks suspicious, 0 → 0.1 → -2.5 → 2.5
        • We could:
          • Drop both messages? Would only require 2 receivers to do error elimination
            • 63.6
            • 63.6, 66.1
            • 63.6, 66.1 [discard 2]
          • Wait until we have 3 simultaneous messages? Would require 3 receivers, but would be error correction.
            • 63.6
            • 63.6, 66.1
            • 63.6, 66.1, 66.1
            • 63.6, 66.1, 66.1
  • Determine a stddev for received messages in the last X time frame. If newly received message is outside that stdev + some error factor (smaller than the current one) then consider the message invalid.