Incorrect State in Prometheus

[tusharsonawanes]

Hi, So I have managed to get blackbox exporter and prometheus running in kubernetes using helm with the commands mentioned below
helm install blackbox prometheus-community/prometheus-blackbox-exporter -n values.yml
helm install prometheus prometheus-community/prometheus -n values.yml

Now, in the values.yml section of prometheus, this is the configuration

extraScrapeConfigs: |
   - job_name: 'prometheus-blackbox-exporter'
     metrics_path: /probe
     params:
       module: [http_2xx]
     static_configs:
       - targets:
         - https://httpstat.us/400
         - https://google.com
     relabel_configs:
       - source_labels: [__address__]
         target_label: __param_target
       - source_labels: [__param_target]
         target_label: instance
       - target_label: __address__
         replacement: blackbox-prometheus-blackbox-exporter:9115

and the important section in values.yml for blackbox exporter (The one which I edited is mentioned below

config:
  modules:
    http_2xx:
      prober: http
      timeout: 5s
      http:
        valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
        follow_redirects: true
        method: GET
        valid_status_codes: []
        preferred_ip_protocol: "ip4"

For the sake of monitoring, I am probing https://httpstat.us/400 which returns 400 when hit or curl'ed.

I have 2 questions / issues

1. What I expected is prometheus should show the state as down since the return http code is not 200. But still it shows as UP. Do I need to make any changes? The expected is, if there is any code except 200, prometheus should show it as DOWN. I will later configure alertmanager to send alerts to channels accordingly.

2. Question - I need to monitor my backend api's using blackbox exporter. All my monitoring pods (Grafana, Prometheus, Blackbox exporter and alertmanager) are in monitoring namespace and the application is in "app" namespace.
   I have a service for backend, frontend and database for the application in the app namespace only. The vision is, for any reason if the api/ service goes down, blackbox should show it as down.

[tusharsonawanes]

Please let me know if I can provide any more files for reference.