heidsoft/cloud-bigdata-book

elk 实践

Opened this issue · 16 comments

heidsoft commented

查看索引

➜  elasticsearch-7.8.1 curl -XGET 'localhost:9200/_cat/indices'
yellow open metricbeat-7.8.1-2020.08.11-000001 Rk__8HYyQWeM0d9xfl6wZA 1 1 49164   0 13.1mb 13.1mb
green  open .apm-custom-link                   zyZJQAw0Q0Glxa6OtaRS8g 1 0     0   0   208b   208b
green  open .kibana_task_manager_1             kRW3wcm_RXmwD_hGOSV9jw 1 0     5  10 10.8kb 10.8kb
green  open .apm-agent-configuration           4_4_VFGAQoGTwx0fig8xHg 1 0     0   0   208b   208b
green  open kibana_sample_data_logs            EGSClSsPSIGHlnDkjxqraQ 1 0 14074   0   11mb   11mb
green  open .async-search                      6A8gMDJASlqMYsB51s451w 1 0     3   0  6.8mb  6.8mb
green  open .kibana_1                          fdk86yC0RtO48RpI4hH6gw 1 0  3176 164  2.1mb  2.1mb
green  open .kibana-event-log-7.8.1-000001     ab62kehLQlOFjRJPBPrUQg 1 0     2   0 10.4kb 10.4kb
yellow open filebeat-7.8.1-2020.08.11-000001   LRoh9ocGSVmc6PdHnPNd6g 1 1 73091   0 12.9mb 12.9mb

➜ elasticsearch-7.8.1

分片状态查看

➜  elasticsearch-7.8.1 curl -XGET 'localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned'
.apm-custom-link                   0 p STARTED
.apm-agent-configuration           0 p STARTED
.async-search                      0 p STARTED
metricbeat-7.8.1-2020.08.11-000001 0 p STARTED
metricbeat-7.8.1-2020.08.11-000001 0 r UNASSIGNED
kibana_sample_data_logs            0 p STARTED
filebeat-7.8.1-2020.08.11-000001   0 p STARTED
filebeat-7.8.1-2020.08.11-000001   0 r UNASSIGNED
.kibana_1                          0 p STARTED
.kibana_task_manager_1             0 p STARTED
ilm-history-2-000001               0 p STARTED
.kibana-event-log-7.8.1-000001     0 p STARTED
➜  elasticsearch-7.8.1

ELK实现日志监控告警

https://www.ctolib.com/docs/sfile/ELKstack-guide-cn/elasticsearch/other/elastalert.html
https://anjia0532.github.io/2017/02/14/elasticsearch-elastalert/
https://media.readthedocs.org/pdf/elastalert/latest/elastalert.pdf
https://anjia0532.github.io/2017/02/14/elasticsearch-elastalert/
https://blog.51cto.com/seekerwolf/2121070
https://www.ctolib.com/docs/sfile/ELKstack-guide-cn/elasticsearch/other/elastalert.html
https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html

heidsoft commented

logstash-plugin

➜ logstash-6.6.0 bin/logstash-plugin install logstash-input-beats
Validating logstash-input-beats
Installing logstash-input-beats
Installation successful
➜ logstash-6.6.0 bin/logstash-plugin install logstash-codec-rubydebug
Validating logstash-codec-rubydebug
Installing logstash-codec-rubydebug
Installation successful

heidsoft commented

ElasticSearch 状态查询Cat接口

https://yq.aliyun.com/articles/676135

heidsoft commented

es深度分页机制

heidsoft commented

es api 测试

curl -X PUT "localhost:9200/test" -H 'Content-Type: application/json' -d'
{
    "settings" : {
        "index" : {
            "number_of_shards" : 1, 
            "number_of_replicas" : 0
        }
    },
    "mappings": {
    	"_doc":{
	    	"properties": { 
			      "title":    { "type": "text"  }, 
			      "name":     { "type": "text"  }, 
			      "age":      { "type": "integer" },
			      "created":  {
			          "type":   "date", 
			          "format": "strict_date_optional_time||epoch_millis"
			        }
		    }
    	}   
   }
}
'

curl -X PUT "localhost:9200/test/_mapping/_doc" -H 'Content-Type: application/json' -d'
{
  "properties": { 
        "title":    { "type": "text"  }, 
        "name":     { "type": "text"  }, 
        "age":      { "type": "integer" },  
        "created":  {
          "type":   "date", 
          "format": "strict_date_optional_time||epoch_millis"
        }
      }
}
'