Prometheus not show any values in http://localhost:1234/metrics
Opened this issue · 10 comments
This is my Pingora server code and I added Prometheus but I can not see any values using http://localhost:1234/metrics URL. How to solve this issue? I can see "my_counter" value increasing by println!("my_counter: {}", MY_COUNTER.get()); code. But nothing show in metrics.
I check Prometheus server run or not,
ss -tuln | grep 1234
tcp LISTEN 0 4096 0.0.0.0:1234 0.0.0.0:* This is my code:
use async_trait::async_trait;
use clap::Parser;
use log::info;
use once_cell::sync::Lazy;
use pingora_core::Result;
use pingora_core::listeners::tls::TlsSettings;
use pingora_core::server::Server;
use pingora_core::server::configuration::Opt;
use pingora_core::services::background::background_service;
use pingora_core::services::listening::Service;
use pingora_core::upstreams::peer::HttpPeer;
use pingora_load_balancing::{LoadBalancer, health_check, selection::RoundRobin};
use pingora_proxy::http_proxy_service;
use pingora_proxy::{ProxyHttp, Session};
use prometheus::{IntCounter, register_int_counter};
use std::{sync::Arc, time::Duration};
pub struct LB(Arc<LoadBalancer<RoundRobin>>); // Load Balancer with Round Robin selection
static MY_COUNTER: Lazy<IntCounter> =
Lazy::new(|| register_int_counter!("my_counter", "my counter").unwrap());
// Create Load Balance Proxy
#[async_trait]
impl ProxyHttp for LB {
// For this small example, we don't need context storage
type CTX = ();
fn new_ctx(&self) -> Self::CTX {}
// Upstream_peer returns the address where the request should be proxied to.
async fn upstream_peer(&self, _session: &mut Session, _ctx: &mut ()) -> Result<Box<HttpPeer>> {
let upstream = self
.0
.select(b"", 256) // hash doesn't matter
.unwrap(); // use the select() method for the LoadBalancer to round-robin across the upstream IPs.
info!("upstream peer is: {:?}", upstream);
// Set SNI to one.one.one.one
let peer = Box::new(HttpPeer::new(
upstream,
false,
"one.one.one.one".to_string(),
)); // Set SNI to one.one.one.one and disable TLS (HTTP) with flase / enable TLS (HTTPS) with true
Ok(peer)
}
// In order for the 1.1.1.1 backends to accept our requests, a host header must be present.
// Adding this header can be done by the upstream_request_filter() callback which modifies the
// request header after the connection to the backends are established and before the request header is sent.
async fn upstream_request_filter(
&self,
_session: &mut Session,
upstream_request: &mut pingora_http::RequestHeader,
_ctx: &mut Self::CTX,
) -> Result<()> {
upstream_request
.insert_header("Host", "one.one.one.one")
.unwrap();
Ok(())
}
async fn logging(
&self,
session: &mut Session,
_e: Option<&pingora_core::Error>,
ctx: &mut Self::CTX,
) {
let response_code = session
.response_written()
.map_or(0, |resp| resp.status.as_u16());
info!(
"{} response code: {response_code}",
self.request_summary(session, ctx)
);
MY_COUNTER.inc();
println!("my_counter: {}", MY_COUNTER.get());
}
}
// RUST_LOG=INFO cargo run --example load_balancer
fn main() {
env_logger::init();
MY_COUNTER.inc();
println!("my_counter: {}", MY_COUNTER.get());
let opt = Opt::parse_args(); // read command line arguments, so cargo run -- -h command "-- -h" to see help
let mut my_server = Server::new(Some(opt)).unwrap(); // create server with command line arguments
my_server.bootstrap();
// All list of array upstreams servers to load balance
let mut upstreams = LoadBalancer::try_from_iter(["127.0.0.1:3000"]).unwrap();
// ===================
// Health Check
// ===================
// We add health check in the background so that the bad server is never selected.
let hc = health_check::TcpHealthCheck::new();
upstreams.set_health_check(hc);
upstreams.health_check_frequency = Some(Duration::from_secs(1));
let background = background_service("health check", upstreams);
let upstreams = background.task();
// ===================
// Load Balance Proxy
// ===================
let mut lb = http_proxy_service(&my_server.configuration, LB(upstreams));
lb.add_tcp("0.0.0.0:6188");
// ===================
// TLS with H2 enabled to load balance proxy
// ===================
let cert_path = format!("{}/certs/cert.pem", env!("CARGO_MANIFEST_DIR"));
let key_path = format!("{}/certs/key.pem", env!("CARGO_MANIFEST_DIR"));
let mut tls_settings = TlsSettings::intermediate(&cert_path, &key_path).unwrap();
tls_settings.enable_h2(); // This work HTTP/2.0 and HTTP/1.1 (Check it)
lb.add_tls_with_settings("0.0.0.0:6189", None, tls_settings);
// ===================
// Prometheus HTTP Service
// ===================
let mut prometheus_service_http = Service::prometheus_http_service();
prometheus_service_http.add_tcp("0.0.0.0:1234");
// ===================
// Run Server
// ===================
my_server.add_service(prometheus_service_http); // add prometheus http service to server
my_server.add_service(lb); // add load balance proxy to server
my_server.add_service(background); // add health check to server
my_server.run_forever(); // spawn all the runtime threads and block the main thread until the server is ready to exit.
}This is my Toml file:
[package]
name = "test_pingora"
version = "0.1.0"
edition = "2024"
[dependencies]
axum = { version = "0.8.4", optional = true }
tokio = { version = "1.44.2", features = ["full"], optional = true }
# pingora = "0.5.0"
pingora-core = { version = "0.5.0", features = ["rustls"], optional = true }
pingora-proxy = { version = "0.5.0", optional = true }
pingora-load-balancing = { version = "0.5.0", optional = true }
pingora-http = { version = "0.5.0", optional = true }
async-trait = { version = "0.1.52", optional = true }
log = { version = "0.4.17", optional = true }
env_logger = { version = "0.11.8", optional = true }
clap = { version = "4.5.39", features = ["derive"] }
prometheus = { version = "0.14.0", optional = true }
once_cell = "1.21.3"
[features]
server = ["axum", "tokio"]
proxy = [
"pingora-proxy",
"pingora-http",
"pingora-load-balancing",
"async-trait",
"log",
"env_logger",
"pingora-core",
"prometheus",
]
[[bin]]
name = "server"
path = "src/server.rs"
required-features = ["server"]
[[bin]]
name = "proxy"
path = "src/proxy.rs"
required-features = ["proxy"]I encounted the same problem, and does pingora have built-in metrics? i used the sameple from the pingora code and export the prometheus scaped server, and accessed that url, but i got nothing.
Any Solution?
from pingora source codes, it seems that pingora doesn't have built-in metrics, users need to add their metrics into the related filters. Instead, pingora only provides a promethus agent.
@hopkings2008 so did you manage to get promethus values? Any code?
@hopkings2008 so did you manage to get promethus values? Any code?
Currently, i didin't get the values exported from promethus provided by pingora.
I've just noticed similar problem after upgrading prometheus crate to 0.14. 0.13 works fine.
Yeah, 0.13 work fine but not for 0.14
Yeah, 0.13 work fine but not for 0.14
after set the prometheus to 0.13, but the problem still exists and there is nothing displayed in the export url by prometheus.
Same here.
If I fetch metrics programmatically
pub fn report_metrics() -> String {
let encoder = TextEncoder::new();
let metric_families = prometheus::gather();
let mut buffer = vec![];
encoder.encode(&metric_families, &mut buffer).unwrap();
String::from_utf8(buffer).unwrap()
}and print it, it works. But nothing via Service::prometheus_http_service.
This is more a dependency conflict than any code incompatibility I think. 0.13 and 0.14 are semver incompatible versions:
This guide uses the terms “major” and “minor” assuming this relates to a “1.0.0” release or later. Initial development releases starting with “0.y.z” can treat changes in “y” as a major release, and “z” as a minor release. “0.0.z” releases are always major changes. This is because Cargo uses the convention that only changes in the left-most non-zero component are considered incompatible.
(from https://doc.rust-lang.org/cargo/reference/semver.html)
You can't patch out semver incompatibilities in Cargo.toml so what happens when you compare the cargo tree output is that the pingora crate gets a different version of the prometheus crate from the rest of your application, hence a different registry, etc.
If you don't want to fix the prometheus version in your application you can pass version = "*" and it will take the version from pingora.