Prometheus Never Prints the Buffer
grippy opened this issue · 9 comments
Been scratching my head at why I'm not seeing metrics sent to Prometheus.
Looks like this strbuf
is created for the metrics but never passed to `self.flush_inner
https://github.com/fralalonde/dipstick/blob/master/src/output/prometheus.rs#L104
https://github.com/fralalonde/dipstick/blob/master/src/output/prometheus.rs#L137
I'm not familiar with the inner workings on this library to submit a patch just yet. If I can figure it out, I'll update this Issue.
I'll look right into it.
Looks like a refactoring's gone wrong somewhere along the way. I've fixed the buffer issue and will be making sure that proper http requests are made. I'll publish a crate update right after that.
@fralalonde +1 I just fixed this locally... I don't know what you had before but this is what I ended up with...
Please note: the prometheus pushgateway is returning 400 w/ a body of "text format parsing error in line 1: unexpected end of input stream\n"
because of the missing '\n' at the end up of the buffer.
fn print(&self, metric: &PrometheusMetric, value: MetricValue, labels: Labels) {
let scaled_value = value / metric.scale;
let value_str = scaled_value.to_string();
let mut buffer = self.buffer.borrow_mut();
// prometheus format be like `http_requests_total{method="post",code="200"} 1027 1395066363000`
buffer.push_str(&metric.prefix);
let labels_map = labels.into_map();
if !labels_map.is_empty() {
buffer.push('{');
let mut i = labels_map.into_iter();
let mut next = i.next();
while let Some((k, v)) = next {
buffer.push_str(&k);
buffer.push_str("=\"");
buffer.push_str(&v);
next = i.next();
if next.is_some() {
buffer.push_str("\",");
} else {
buffer.push('"');
}
}
buffer.push_str("} ");
} else {
buffer.push(' ');
}
buffer.push_str(&value_str);
buffer.push('\n');
if buffer.len() > BUFFER_FLUSH_THRESHOLD {
metrics::PROMETHEUS_OVERFLOW.mark();
warn!(
"Prometheus Buffer Size Exceeded: {}",
BUFFER_FLUSH_THRESHOLD
);
let _ = self.flush_inner(buffer);
} else if !self.is_buffered() {
if let Err(e) = self.flush_inner(buffer) {
debug!("Could not send to Prometheus {}", e)
}
}
}
I just released v0.7.7, let me know how it runs... I'm happy to have someone helping me test Prometheus!
Make that 0.7.8...
@fralalonde One more issue if you didn't fix it:
We should change the http call to post
(instead of get
which isn't supported from what I can tell).
https://github.com/prometheus/pushgateway/blob/master/README.md#put-method
https://github.com/fralalonde/dipstick/blob/master/src/output/prometheus.rs#L163
I'll try out your new release! Thanks for the fix.
I might have some docs coming your way. I've been playing around with using the Proxy => AtomicBucket => MultiOutput pipelines. The kicker here is you need to you use flush otherwise the Prometheus print
code panics trying to use self.buffer.borrow_mut()
if you hammer the pushgateway with too many metrics.
It took me a little while to figure out. Might as well pass it on.
Btw, still seeing this error in the pushgateway
response:
TRACE dipstick::output::prometheus > 400 "text format parsing error in line 1: unexpected end of input stream\n"
See the PR here:
#71
I merged your PR, made some edits to the handbook regarding Proxy and published a new version. The new example you included is pretty nice too. I should document the other examples the same way.
This looks good! Thanks for the quick turnaround.