Riemann writes not all messages to InfluxDB

[Mahabharata871]

Hello.
Installed a bunch of Riemann version 0.3.1 and Influx version 1.8

I accept messages from web servers to Riemann, process them and add them to InfluxDB.
But comparing the number of records in the Riemann log and in the Database, I noticed a difference in the number of messages, i.e. there are more entries in the log than in the database.

Please, tell me what to look for and what to read?
Riemann config below:

`(logging/init {:file "/var/log/riemann/riemann.log"})

(let [host "0.0.0.0" port 5555]
(tcp-server {:host host})
(udp-server {:host host})
(ws-server {:host host})
)

(periodically-expire 5)

(def influxdb-creds {:host "localhost"
:port 8086
:username "username"
:password "password"
:db "riemann"
})

(def influx
(batch 100 1/10
(async-queue! :agg {:queue-size 1000
:core-pool-size 1
:max-pool-size 4
:keep-alive-time 60000}
(influxdb influxdb-creds))))

(require '[cheshire.core :as json])

(defn safe-json [^String s]
(try
(json/parse-string s true)
(catch Exception _ nil)))

(defn restore-payload [event]
(update-in event [:payload] safe-json))

(defn app-event? [e]
(some-> e restore-payload :payload :application))
(let [index (index)]
(streams
(default :ttl 60
index
(where (service "nginx_json")
(smap (fn [ev] (assoc ev :service "RPS" :metric 1 :payload 1))
(rate 1 influx)
#(info (:time %) (:host %) (:service %) (:metric %))
)
))))
`

[sanel]

Probably because log accepts more events than (rate) is allowed to process. You are limiting (rate) to every second.

If you want to validate that log output matches influx inputs, use this (info and influx are in rate expression).

...
(rate 1
  #(info (:time %) (:host %) (:service %) (:metric %))
  influx)))))

[jamtur01]

Please re-open if you still have questions.