Kibana - Readiness probe failed: Error: Got HTTP code 000 but expected a 200

Question

Kibana - Readiness probe failed: Error: Got HTTP code 000 but expected a 200

iamaverrick opened this issue 2 years ago · 0 comments

Chart version:
8.5.1

Kubernetes version:

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8+vmware.1", GitCommit:"d797572df69c3951e4e8d495bf7720b594fd1c43", GitTreeState:"clean", BuildDate:"2022-03-21T23:17:28Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.25) and server (1.22) exceeds the supported minor version skew of +/-1

Kubernetes provider: E.g. GKE (Google Kubernetes Engine)
Vmware Tanzu K8s

Helm Version:
version.BuildInfo{Version:"v3.9.4", GitCommit:"dbc6d8e20fe1d58d50e6ed30f09a04a77e4c68db", GitTreeState:"clean", GoVersion:"go1.19"}

helm get release output

e.g. helm get elasticsearch (replace elasticsearch with the name of your helm release)

Be careful to obfuscate every secrets (credentials, token, public IP, ...) that could be visible in the output before copy-pasting.

If you find some secrets in plain text in helm get release output you should use Kubernetes Secrets to managed them is a secure way (see Security Example).

Output of helm get release

Describe the bug:

kibana deployed just fine and it seems to be running as expected. error comes in while trying to use TLS/SSl termination. when kubectl describe is used we see this in events
Warning Unhealthy 3m2s (x53759 over 5d11h) kubelet Readiness probe failed: Error: Got HTTP code 000 but expected a 200
also page returns a 503 Service Temporarily Unavailable error nginx.

Steps to reproduce:
my deployed kibana config

elasticsearchHosts: "https://elasticsearch:9200"
elasticsearchCertificateSecret: elasticsearch-certs
elasticsearchCertificateAuthoritiesFile: ca.crt
elasticsearchCredentialSecret: elastic-credentials

extraEnvs:
  - name: "NODE_OPTIONS"
    value: "--max-old-space-size=1800"
  - name: "KIBANA_ENCRYPTION_KEY"
    valueFrom:
      secretKeyRef:
        name: kibana-es-token
        key: token


protocol: https

kibanaConfig:
  kibana.yml: |
    server.publicBaseUrl: https://kibana.example.com
    xpack.reporting.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    xpack.encryptedSavedObjects.encryptionKey: ${KIBANA_ENCRYPTION_KEY}

#    server.ssl.enabled: true
#    server.ssl.key: /usr/share/kibana/config/certs/tls.key
#    server.ssl.certificate: /usr/share/kibana/config/certs/tls.crt

httpPortName: https

ingress:
  enabled: true
  className: nginx
  pathtype: ImplementationSpecific
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt"
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/http2-push-preload: 'true'
    nginx.ingress.kubernetes.io/proxy-ssl-verify: 'on'
  hosts:
    - host: kibana.example.com
      paths:
        - path: /
  tls:
    - secretName: kibana-tls
      hosts:
        - kibana.example.com

Expected behavior:
work with ssl/tls termination for production use.

Provide logs and/or server output (if relevant):

[2022-12-10T05:20:28.209+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2022-12-10T05:20:55.599+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
[2022-12-10T05:20:55.623+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
[2022-12-10T05:20:55.877+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
[2022-12-10T05:20:55.995+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2022-12-10T05:20:56.130+00:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2022-12-10T05:20:56.857+00:00][INFO ][plugins-system.standard] Setting up [125] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,guidedOnboarding,fieldFormats,expressions,dataViews,embeddable,uiActionsEnhanced,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,lists,files,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,telemetry,licenseManagement,eventLog,actions,stackConnectors,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,aiops,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,eventAnnotation,dataViewFieldEditor,triggersActionsUi,transform,stackAlerts,ruleRegistry,discover,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,observability,osquery,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
[2022-12-10T05:20:56.900+00:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: ed74d27b-5342-42e8-addb-9bfeb1458ab1
[2022-12-10T05:20:57.199+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2022-12-10T05:20:57.301+00:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2022-12-10T05:20:57.332+00:00][INFO ][plugins.encryptedSavedObjects] Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: 260i43g4a9oTEF8UOI5/F+072ek8rj+hK2+QY+EEwBc=
[2022-12-10T05:20:57.717+00:00][WARN ][plugins.reporting.config] Found 'server.host: "0.0.0.0"' in Kibana configuration. Reporting is not able to use this as the Kibana server hostname. To enable PNG/PDF Reporting to work, 'xpack.reporting.kibanaServer.hostname: localhost' is automatically set in the configuration. You can prevent this message by adding 'xpack.reporting.kibanaServer.hostname: localhost' in kibana.yml.
[2022-12-10T05:20:57.897+00:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
[2022-12-10T05:20:58.022+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2022-12-10T05:21:00.046+00:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 20.04 OS. Automatically enabling Chromium sandbox.
[2022-12-10T05:21:01.500+00:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2022-12-10T05:21:01.502+00:00][INFO ][savedobjects-service] Starting saved objects migrations
[2022-12-10T05:21:01.814+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 105ms.
[2022-12-10T05:21:01.824+00:00][INFO ][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 120ms.
[2022-12-10T05:21:01.829+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 15ms.
[2022-12-10T05:21:01.833+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 9ms.
[2022-12-10T05:21:01.845+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 16ms.
[2022-12-10T05:21:01.852+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 19ms.
[2022-12-10T05:21:01.855+00:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 10ms.
[2022-12-10T05:21:01.859+00:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 7ms.
[2022-12-10T05:21:01.939+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 84ms.
[2022-12-10T05:21:02.352+00:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 413ms.
[2022-12-10T05:21:02.353+00:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 644ms
[2022-12-10T05:21:02.371+00:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 512ms.
[2022-12-10T05:21:03.083+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
[2022-12-10T05:21:03.820+00:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 1449ms.
[2022-12-10T05:21:03.821+00:00][INFO ][savedobjects-service] [.kibana] Migration completed after 2117ms
[2022-12-10T05:21:03.840+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
[2022-12-10T05:21:03.844+00:00][INFO ][plugins-system.standard] Starting [125] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licenseApiGuard,usageCollection,taskManager,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,share,screenshotMode,banners,newsfeed,guidedOnboarding,fieldFormats,expressions,dataViews,embeddable,uiActionsEnhanced,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,spaces,security,lists,files,encryptedSavedObjects,cloud,snapshotRestore,screenshotting,telemetry,licenseManagement,eventLog,actions,stackConnectors,console,bfetch,data,watcher,reporting,fileUpload,ingestPipelines,alerting,aiops,unifiedSearch,unifiedFieldList,savedSearch,savedObjects,graph,savedObjectsTagging,savedObjectsManagement,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,eventAnnotation,dataViewFieldEditor,triggersActionsUi,transform,stackAlerts,ruleRegistry,discover,fleet,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,discoverEnhanced,visualizations,canvas,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeHeatmap,visTypeMarkdown,dashboard,dashboardEnhanced,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,lens,maps,dataVisualizer,cases,timelines,sessionView,kubernetesSecurity,observability,osquery,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,visTypeGauge,dataViewManagement]
[2022-12-10T05:21:07.938+00:00][INFO ][plugins.monitoring.monitoring] config sourced from: production cluster
[2022-12-10T05:21:12.808+00:00][INFO ][http.server.Kibana] http server running at http://0.0.0.0:5601
[2022-12-10T05:21:12.914+00:00][INFO ][status] Kibana is now degraded
[2022-12-10T05:21:13.420+00:00][INFO ][plugins.monitoring.monitoring.kibana-monitoring] Starting monitoring stats collection
[2022-12-10T05:21:13.421+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2022-12-10T05:21:13.727+00:00][INFO ][plugins.ruleRegistry] Installed common resources shared between all indices
[2022-12-10T05:21:13.728+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.uptime.alerts
[2022-12-10T05:21:13.729+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-security.alerts
[2022-12-10T05:21:13.729+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .preview.alerts-security.alerts
[2022-12-10T05:21:13.730+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.logs.alerts
[2022-12-10T05:21:13.731+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.metrics.alerts
[2022-12-10T05:21:13.731+00:00][INFO ][plugins.ruleRegistry] Installing resources for index .alerts-observability.apm.alerts
[2022-12-10T05:21:13.760+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.metrics.alerts
[2022-12-10T05:21:13.761+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.uptime.alerts
[2022-12-10T05:21:13.763+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.logs.alerts
[2022-12-10T05:21:13.765+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-security.alerts
[2022-12-10T05:21:13.766+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .alerts-observability.apm.alerts
[2022-12-10T05:21:13.953+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .preview.alerts-security.alerts
[2022-12-10T05:21:14.148+00:00][INFO ][plugins.fleet] Fleet setup completed
[2022-12-10T05:21:14.158+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: scheduled with interval 1h
[2022-12-10T05:21:14.165+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
[2022-12-10T05:21:14.235+00:00][INFO ][plugins.synthetics] Installed synthetics index templates
[2022-12-10T05:21:16.546+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: No ML saved objects in need of synchronization
[2022-12-10T05:21:18.955+00:00][INFO ][status] Kibana is now available (was degraded)

Be careful to obfuscate every secrets (credentials, token, public IP, ...) that could be visible in the output before copy-pasting

Any additional context:

service only works when using HTTP and ssl/tls is off. I'm not sure what I'm doing wrong here or if this is a BUG.