istio-request-total is missed in version 1.78.0 but in 1.70.0 it works

[AryaCherryLiu]

I am using version 1.78.0

there is my configuration:

values:
   nameOverride: splunk-otel
   clusterName: oc-${ENVIRONMENT}${ENVIRONMENT_SUFFIX}
   environment: oc-${ENVIRONMENT}
   extraAttributes:
     fromLabels:
     - key: app
       tag_name: label_app
       from: pod
     - key: release
       tag_name: label_release
       from: pod
     - key: type
       tag_name: label_type
       from: pod
     custom:
     - name: cluster_codename
       value: ${CLUSTER_NAME}
     - name: cluster_region
       value: ${AWS_REGION}
     - name: project
       value: <project>
   splunkPlatform:
     endpoint: https://${SPLUNK_HOST_NAME}:8088/services/collector
     insecureSkipVerify: true
     timeout: 30s
     index: infra-${ENVIRONMENT}
     metricsEnabled: false
     metricsIndex: infra-${ENVIRONMENT}-metrics
     fieldNameConvention:
       renameFieldsSck: true
       keepOtelConvention: false
   logsEngine: otel
   splunkObservability:
     realm: us1
   cloudProvider: aws
   distribution: eks
   gateway:
     enabled: true
     replicaCount: ${SPLUNK_OTEL_GATEWAY_REPLICAS}
     resources:
       requests:
         cpu: ${SPLUNK_OTEL_GATEWAY_REQUEST_CPU}
         memory: ${SPLUNK_OTEL_GATEWAY_REQUEST_MEMORY}
       limits:
         cpu: ${SPLUNK_OTEL_GATEWAY_LIMIT_CPU}
         memory: ${SPLUNK_OTEL_GATEWAY_LIMIT_MEMORY}
   clusterReceiver:
     k8sEventsEnabled: true
     config:
       exporters:
         signalfx:
           include_metrics:
             - metric_name: k8s.hpa.max_replicas
             - metric_name: k8s.hpa.current_replicas
             - metric_name: k8s.hpa.min_replicas
             - metric_name: k8s.hpa.desired_replicas
             - metric_name: k8s.cronjob.active_jobs
       receivers:
         k8s_cluster:
           node_conditions_to_report:
             - "Ready"
             - "MemoryPressure"
           allocatable_types_to_report: 
             - "cpu"
             - "memory"
     resources:
       requests:
         cpu: ${SPLUNK_OTEL_RECEIVER_REQUEST_CPU}
         memory: ${SPLUNK_OTEL_RECEIVER_REQUEST_MEMORY}
       limits:
         cpu: ${SPLUNK_OTEL_RECEIVER_LIMIT_CPU}
         memory: ${SPLUNK_OTEL_RECEIVER_LIMIT_MEMORY}
   agent:
     config:
       extensions:
         k8s_observer:
           auth_type: serviceAccount
           observe_pods: true
       receivers:
         receiver_creator:
           receivers:
             prometheus_simple:
                 rule: type == "pod" && annotations["prometheus.io/scrape"] == "true" && annotations["prometheus.io/port"] == "15020"
                 config:
                   metrics_path: "/stats/prometheus"
                   endpoint: '`endpoint`:`"prometheus.io/port" in annotations ? annotations["prometheus.io/port"] : 15020`'
                   collection_interval: 15s
             smartagent/nginx:
               rule: type == "pod" && labels["app"] startsWith "rp-"
               config:
                 type: collectd/nginx
                 port: 8080
                 host: "`endpoint`"
       processors:
         filter/istio_access_metrics:
           metrics:
             exclude:
               match_type: regexp
               metric_names:
                 - istio_agent.*
       service:
         pipelines:
           metrics:
             processors:
               - filter/istio_access_metrics
               - memory_limiter
               - batch
   autodetect:
     istio: true
     prometheus: true

when I check Splunk Observability, I can't find any metric for istio_request_total, but for others, they stay there

after I change the version to 1.70.0 the istio_request_total metric apeared in Splunk Observability

I have no idea about this issue. anyone can help me figure it out

[atoulme]

You are using 0.78.0 which introduced a breaking change in naming conventions for prometheus receiver, and was reverted in 0.79.0. Please upgrade to the latest collector to fix this issue.