Getting API Error Error Fetching Indices
Closed this issue · 19 comments
Hi Daichi,
I tried to deploy praeco server using your helm chart i am able to get the UI of praeco but it is not able to connect my elasticsearch which is running on the same cluster.
i deployed elastalert also using - https://github.com/helm/charts/tree/master/stable/elastalert
Please help me on this.
Hi, @suresh447 Thanks for using my helm chart.
You don't have to deploy elastalert yourself, because elastalert is automatiically deployed by dependencies in requrement.yaml.
dependencies:
- name: elastalert
version: 0.1.1
repository: https://daichi703n.github.io/charts/stable
ElastAlert is deployed automatically, isn't it?
For connecting to your Elasticsearch, please configure your Elasticsearch url into vars.yml and deploy with helm install --name my-release daichi703n/praeco -v vars.yml.
sample vars.yml is here.
https://github.com/daichi703n/praeco-helm/blob/master/vars.yml.example
I succussfully ran the praeco server it is not allowing to create rules it is giving me the error like
i checked the logs of elastalert logs and these are the logs
ProcessController: Elastic Version: 7.6.1
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
Index elastalert_status already exists. Skipping index creation.
02:23:21.774Z INFO elastalert-server: ProcessController: Index create exited with code 0
02:23:21.774Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
02:23:21.786Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 36)
02:23:21.788Z INFO elastalert-server: Server: Server listening on port 3030
02:23:21.789Z INFO elastalert-server: Server: Websocket listening on port 3333
02:23:21.790Z INFO elastalert-server: Server: Server started
02:23:22.501Z ERROR elastalert-server:
ProcessController: WARNING:elastalert:Enabled rules are: []
02:23:22.502Z ERROR elastalert-server:
ProcessController: WARNING:elastalert:Disabled rules are: []
02:23:23.798Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:23:25.199Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:23:33.789Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:23:35.198Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:23:43.790Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:23:45.199Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:23:53.790Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:23:55.199Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:03.790Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:05.199Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:13.790Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:15.199Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:22.558Z ERROR elastalert-server:
ProcessController: WARNING:elastalert:Enabled rules are: []
02:24:22.558Z ERROR elastalert-server:
ProcessController: WARNING:elastalert:Disabled rules are: []
02:24:23.790Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:25.199Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:33.789Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:35.199Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:43.790Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:45.199Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:53.789Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:24:55.199Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:25:03.789Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:25:05.199Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:25:13.790Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:25:15.199Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
02:25:22.601Z ERROR elastalert-server:
ProcessController: WARNING:elastalert:Enabled rules are: []
02:25:22.601Z ERROR elastalert-server:
ProcessController: WARNING:elastalert:Disabled rules are: []
This error seems to be caused by volume mount issue.
Is your persistent volume claim created properly?
Yes PVC created properly only please find these are the below PVC
elasticsearch-master-elasticsearch-master-0 Bound pvc-7507a93a-5eb5-11ea-a70e-ba38edfe70a2 30Gi RWO default 3d22h
praeco-elastalert Bound pvc-9145a032-60e3-11ea-a70e-ba38edfe70a2 1Gi RWO default 27h
Umm...
Please let me know your Praeco-ElastAlert connectivity. Does /api, /api-ws return as below?
{"name":"elastalert-server","port":3030,"version":"0.0.14"}
Upgrade Required
yes @daichi703n i am getting the same response
{"name":"elastalert-server","port":3030,"version":"0.0.14"}
Upgrade Required
Good.
Elasticsearch you want to connect is built by elastic/elasticsearch helm chart? (According to your PVCs)
Please show me your vars.yml. If you are using elastic/elasticsearch helm chart, you don't have to specify elasticsearch url in vars (able to use default).
yes i have installed elasticsearch using elastic/elasticsearch helm chart only and my vars.yml file is
elastalert:
elasticsearch:
host: "30.195.xx.xx"
port: 9200
useSsl: false
#username: ""
#password: ""
#slack_webhook_url: https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXX
#extraConfigOptions:
slack_ignore_ssl_errors: true
praeco:
schema: http
external_host: praeco
port: 8080
if i use elasticsearch url as default in vars.yml file it is throwing me the error
API Error
Error fetching indices
elastic/elasticsearch creates service for accessing elasticsearch with http://elasticsearch-master:9200.
Please kubectl exec into elastalert and curl http://elasticsearch-master:9200, does it work?
I am able to get the elasticsearch response using my IP address but not by elasticsearch-master
/opt/elastalert-server $ curl http://elasticsearch-master:9200
curl: (6) Could not resolve host: elasticsearch-master
/opt/elastalert-server $ curl http://30.195.xx.xxx:9200
{
"name" : "elasticsearch-master-0",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "cGv_28sSRoKVbrI43xecXg",
"version" : {
"number" : "7.6.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "aa751e09be0a5072e8570670309b1f12348f023b",
"build_date" : "2020-02-29T00:15:25.529771Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
Thanks.
Is config.yaml in elastalert rendered properly?
$ kubectl -n elastic exec -it praeco-elastalert-654c6979fc-mxwlz sh
/opt/elastalert-server $ cat /opt/elastalert/config.yaml
---
rules_folder: /opt/elastalert/rules
# scan_subdirectories: false
run_every:
minutes: 1
buffer_time:
minutes: 15
es_host: elasticsearch-master #<-should be 30.195.xx.xxx
es_port: 9200
writeback_index: elastalert_status
use_ssl: false
verify_certs: true
alert_time_limit:
minutes: 2880
I configured correctly only but still i am getting the same error
rules_folder: /opt/elastalert/rules
scan_subdirectories: false
run_every:
minutes: 1
buffer_time:
minutes: 15
es_host: 30.195.XX.XXX
es_port: 9200
writeback_index: elastalert_status
use_ssl: false
verify_certs: true
alert_time_limit:
Rule not found
What is your /opt/elastalert/rules's permission?
/opt/elastalert-server $ ls -la /opt/elastalert/rules/
total 8
drwxrwxrwx 2 root root 46 Feb 24 10:47 .
drwxr-xr-x 1 node node 24 Aug 27 2019 ..
-rw-r--r-- 1 root root 246 Feb 24 10:57 BaseRule.config
-rw-r--r-- 1 node node 602 Feb 24 10:55 test.yaml
If it is root 644 ,I may have failed to set PVC permissions. That is my mistake in creating Helm-Chart...
My /opt/elastalert/rules's permission? is
total 172
drwxr-xr-x 1 node node 4096 Aug 27 2019 .
drwxr-xr-x 1 node node 4096 Aug 25 2019 ..
-rw-r--r-- 1 node node 216 Aug 25 2019 .editorconfig
-rw-r--r-- 1 node node 160 Aug 25 2019 .gitignore
-rw-r--r-- 1 node node 718 Aug 25 2019 .pre-commit-config.yaml
-rw-r--r-- 1 node node 574 Aug 25 2019 .secrets.baseline
-rw-r--r-- 1 node node 1262 Aug 25 2019 .travis.yml
-rw-r--r-- 1 node node 264 Aug 25 2019 Dockerfile-test
-rw-r--r-- 1 node node 11359 Aug 25 2019 LICENSE
-rw-r--r-- 1 node node 528 Aug 25 2019 Makefile
-rw-r--r-- 1 node node 15705 Aug 25 2019 README.md
drwxr-xr-x 1 node node 4096 Aug 25 2019 build
-rw-r--r-- 1 node node 10707 Aug 25 2019 changelog.md
-rw-r--r-- 1 root root 259 Mar 9 09:50 config.yaml
-rw-r--r-- 1 node node 3321 Aug 25 2019 config.yaml.example
drwxr-xr-x 1 node node 4096 Aug 25 2019 dist
-rw-r--r-- 1 node node 261 Aug 25 2019 docker-compose.yml
drwxrwxrwx 1 node node 4096 Aug 25 2019 docs
drwxrwxrwx 1 node node 4096 Mar 9 09:51 elastalert
drwxr-xr-x 1 node node 4096 Aug 25 2019 elastalert.egg-info
drwxr-xr-x 1 node node 4096 Aug 27 2019 elastalert_modules
drwxrwxrwx 1 node node 4096 Aug 25 2019 example_rules
-rw-r--r-- 1 node node 74 Aug 25 2019 pytest.ini
-rw-r--r-- 1 node node 107 Aug 25 2019 requirements-dev.txt
-rw-r--r-- 1 node node 449 Aug 25 2019 requirements.txt
drwxr-xr-x 2 root root 4096 Mar 9 09:51 rule_templates
drwxr-xr-x 2 root root 4096 Mar 9 09:51 rules
drwxr-xr-x 1 node node 4096 Aug 27 2019 server_data
-rw-r--r-- 1 node node 100 Aug 25 2019 setup.cfg
-rw-r--r-- 1 node node 1711 Aug 25 2019 setup.py
-rw-r--r-- 1 node node 780 Aug 25 2019 supervisord.conf.example
drwxrwxrwx 1 node node 4096 Aug 25 2019 tests
-rw-r--r-- 1 node node 606 Aug 25 2019 tox.ini
Sorry, I have made a mistake in how to create PVC permissions ...
drwxr-xr-x 2 root root 4096 Mar 9 09:51 rule_templates
drwxr-xr-x 2 root root 4096 Mar 9 09:51 rules
Should be
drwxrwxrwx 2 root root 4096 Mar 9 09:51 rule_templates
drwxrwxrwx 2 root root 4096 Mar 9 09:51 rules
ElastAlert writes rule files with node user, therefore permission error will occur.
I'll check how to modify permission on initial deploy. At a moment, you can modify these by running chmod 777 /opt/elastalert/rule_template and chmod 777 /opt/elastalert/rules, or chown node:node /opt/elastalert/rule_template and chwon node:node /opt/elastalert/rules in elastalert container. (I know this is not good way...)
This is not the way to change filesystem permissions i think it is not allowing me to perform this commands inside the container.
/opt/elastalert $ chown node:node /opt/elastalert/rule_templates
chown: /opt/elastalert/rule_templates: Operation not permitted
ElastAlert container does not accept elevate to root... sorry...
I’ll fix it, but I can’t use developing environment few days. Please wait to fix it.
Hi, @suresh447
Sorry for late response.
I released bugfix version (0.1.2). Please helm repo update and deploy it.
I fixed issue with creating initContainers. It seems to fix permission issue.
daichi703n/elastalert-helm@e281f34
On my rancher k3s env, pvc was created as 777, therefore I couldn't detect the issue. Thanks for your testing.