No dashboards actually show up after running script
Closed this issue · 11 comments
alexanderfichel commented
I am using AWS managed service and I get:
$ ./load.sh -url "http://search-abc-123.us-themoon-99.es.amazonaws.com"
{"Message":"User: anonymous is not authorized to perform: es:ESHttpPut on resource: arn:aws:es:us-west-2:1235573471613174:domain/windows-apache-cluster/.kibana/visualization/PgSQL-Errors"}
So it is not using AWS signed requests? Any way you guys could maybe add that feature?
Even if I put open domain policy: I get false for created so nothing gets created...
Loading index pattern filebeat-*:
{"_index":".kibana","_type":"index-pattern","_id":"filebeat-*","_version":3,"created":false}
Loading index pattern packetbeat-*:
{"_index":".kibana","_type":"index-pattern","_id":"packetbeat-*","_version":3,"created":false}
Loading index pattern topbeat-*:
{"_index":".kibana","_type":"index-pattern","_id":"topbeat-*","_version":3,"created":false}
Loading index pattern winlogbeat-*:
{"_index":".kibana","_type":"index-pattern","_id":"winlogbeat-*","_version":3,"created":false}
tsg commented
I marked it as an enhancement request. I'm not too familiar with the AWS search service, is there no way of using basic auth or an HTTP header for auth?
alexanderfichel commented
Well, after I set to open domain policy, the authorization error went away, but all "created" are false
tsg commented
created false just means that the doc was already there and it got overwritten. How do you check if the dashboards are loaded?
alexanderfichel commented
I just go to Kibana dashboards and refresh the browser, shouldn't they just appear there?
dedemorton commented
Did you open the saved dashboards as described in the doc? https://www.elastic.co/guide/en/beats/libbeat/current/view-kibana-dashboards.html
dedemorton commented
To clarify my previous comment: Did you go to the Dashboard page and click the Load Saved Dashboard icon? You need to do that and then select Packetbeat Dashboard from the list, and you'll be able to select from the Beats dashboards.
alexanderfichel commented
Yes, I did that. Actually a similar issue can be seen in stackoverflow:
alexanderfichel commented
I completely wiped out my indices and ran command:
./load.sh -url "search-123.us-west-2.es.amazonaws.com" -index topbeat-03-30-2016
I get good output:
[root@ip-10-218-249-94 beats-dashboards]# ./load.sh -url "search-new-test-123.us-west-2.es.amazonaws.com" -index topbeat-03-30-2016
Loading dashboards to search-new-test-123.us-west-2.es.amazonaws.com in topbeat-03-30-2016
{"acknowledged":true}{"acknowledged":true}Loading search Cache-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Cache-transactions","_version":1,"created":true}
Loading search DB-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"DB-transactions","_version":1,"created":true}
Loading search Default-Search:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Default-Search","_version":1,"created":true}
Loading search Errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Errors","_version":1,"created":true}
Loading search Filesystem-stats:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Filesystem-stats","_version":1,"created":true}
Loading search HTTP-errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"HTTP-errors","_version":1,"created":true}
Loading search MongoDB-errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"MongoDB-errors","_version":1,"created":true}
Loading search MongoDB-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"MongoDB-transactions","_version":1,"created":true}
Loading search MongoDB-transactions-with-write-concern-0:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"MongoDB-transactions-with-write-concern-0","_version":1,"created":true}
Loading search MySQL-errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"MySQL-errors","_version":1,"created":true}
Loading search MySQL-Transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"MySQL-Transactions","_version":1,"created":true}
Loading search Packetbeat-Search:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Packetbeat-Search","_version":1,"created":true}
Loading search PgSQL-errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"PgSQL-errors","_version":1,"created":true}
Loading search PgSQL-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"PgSQL-transactions","_version":1,"created":true}
Loading search Proc-stats:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Proc-stats","_version":1,"created":true}
Loading search RPC-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"RPC-transactions","_version":1,"created":true}
Loading search System-stats:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"System-stats","_version":1,"created":true}
Loading search Thrift-errors:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Thrift-errors","_version":1,"created":true}
Loading search Thrift-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Thrift-transactions","_version":1,"created":true}
Loading search Web-transactions:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Web-transactions","_version":1,"created":true}
Loading search Winlogbeat-Search:
{"_index":"topbeat-03-30-2016","_type":"search","_id":"Winlogbeat-Search","_version":1,"created":true}
Loading visualization Cache-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Cache-transactions","_version":1,"created":true}
Loading visualization Client-locations:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Client-locations","_version":1,"created":true}
Loading visualization CPU-usage:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"CPU-usage","_version":1,"created":true}
Loading visualization CPU-usage-per-process:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"CPU-usage-per-process","_version":1,"created":true}
Loading visualization DB-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"DB-transactions","_version":1,"created":true}
Loading visualization Disk-utilization-over-time:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Disk-utilization-over-time","_version":1,"created":true}
Loading visualization Errors-count-over-time:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Errors-count-over-time","_version":1,"created":true}
Loading visualization Errors-vs-successful-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Errors-vs-successful-transactions","_version":1,"created":true}
Loading visualization Event-Levels:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Event-Levels","_version":1,"created":true}
Loading visualization HTTP-codes-for-the-top-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"HTTP-codes-for-the-top-queries","_version":1,"created":true}
Loading visualization HTTP-error-codes-evolution:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"HTTP-error-codes-evolution","_version":1,"created":true}
Loading visualization HTTP-error-codes:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"HTTP-error-codes","_version":1,"created":true}
Loading visualization Latency-histogram:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Latency-histogram","_version":1,"created":true}
Loading visualization Levels:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Levels","_version":1,"created":true}
Loading visualization Memory-usage:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Memory-usage","_version":1,"created":true}
Loading visualization Memory-usage-per-process:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Memory-usage-per-process","_version":1,"created":true}
Loading visualization MongoDB-commands:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-commands","_version":1,"created":true}
Loading visualization MongoDB-errors:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-errors","_version":1,"created":true}
Loading visualization MongoDB-errors-per-collection:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-errors-per-collection","_version":1,"created":true}
Loading visualization MongoDB-in-slash-out-throughput:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-in-slash-out-throughput","_version":1,"created":true}
Loading visualization MongoDB-response-times-and-count:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-response-times-and-count","_version":1,"created":true}
Loading visualization MongoDB-response-times-by-collection:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MongoDB-response-times-by-collection","_version":1,"created":true}
Loading visualization Most-frequent-MySQL-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Most-frequent-MySQL-queries","_version":1,"created":true}
Loading visualization Most-frequent-PgSQL-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Most-frequent-PgSQL-queries","_version":1,"created":true}
Loading visualization MySQL-Errors:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MySQL-Errors","_version":1,"created":true}
Loading visualization MySQL-Methods:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MySQL-Methods","_version":1,"created":true}
Loading visualization MySQL-Reads-vs-Writes:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MySQL-Reads-vs-Writes","_version":1,"created":true}
Loading visualization Mysql-response-times-percentiles:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Mysql-response-times-percentiles","_version":1,"created":true}
Loading visualization MySQL-throughput:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"MySQL-throughput","_version":1,"created":true}
Loading visualization Navigation:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Navigation","_version":1,"created":true}
Loading visualization Number-of-Events:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Number-of-Events","_version":1,"created":true}
Loading visualization Number-of-Events-Over-Time-By-Event-Log:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Number-of-Events-Over-Time-By-Event-Log","_version":1,"created":true}
Loading visualization Number-of-MongoDB-transactions-with-writeConcern-w-equal-0:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Number-of-MongoDB-transactions-with-writeConcern-w-equal-0","_version":1,"created":true}
Loading visualization PgSQL-Errors:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"PgSQL-Errors","_version":1,"created":true}
Loading visualization PgSQL-Methods:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"PgSQL-Methods","_version":1,"created":true}
Loading visualization PgSQL-Reads-vs-Writes:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"PgSQL-Reads-vs-Writes","_version":1,"created":true}
Loading visualization PgSQL-response-times-percentiles:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"PgSQL-response-times-percentiles","_version":1,"created":true}
Loading visualization PgSQL-throughput:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"PgSQL-throughput","_version":1,"created":true}
Loading visualization Process-status:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Process-status","_version":1,"created":true}
Loading visualization Reads-versus-Writes:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Reads-versus-Writes","_version":1,"created":true}
Loading visualization Response-times-percentiles:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Response-times-percentiles","_version":1,"created":true}
Loading visualization Response-times-repartition:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Response-times-repartition","_version":1,"created":true}
Loading visualization RPC-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"RPC-transactions","_version":1,"created":true}
Loading visualization Servers:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Servers","_version":1,"created":true}
Loading visualization Slowest-MySQL-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Slowest-MySQL-queries","_version":1,"created":true}
Loading visualization Slowest-PgSQL-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Slowest-PgSQL-queries","_version":1,"created":true}
Loading visualization Slowest-Thrift-RPC-methods:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Slowest-Thrift-RPC-methods","_version":1,"created":true}
Loading visualization Sources:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Sources","_version":1,"created":true}
Loading visualization System-load:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"System-load","_version":1,"created":true}
Loading visualization Thrift-requests-per-minute:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Thrift-requests-per-minute","_version":1,"created":true}
Loading visualization Thrift-response-times-percentiles:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Thrift-response-times-percentiles","_version":1,"created":true}
Loading visualization Thrift-RPC-Errors:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Thrift-RPC-Errors","_version":1,"created":true}
Loading visualization Top-10-HTTP-requests:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-10-HTTP-requests","_version":1,"created":true}
Loading visualization Top-10-memory-consumers:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-10-memory-consumers","_version":1,"created":true}
Loading visualization Top-10-processes-by-total-CPU-usage:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-10-processes-by-total-CPU-usage","_version":1,"created":true}
Loading visualization Top-Event-IDs:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-Event-IDs","_version":1,"created":true}
Loading visualization Top-processes:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-processes","_version":1,"created":true}
Loading visualization Top-slowest-MongoDB-queries:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-slowest-MongoDB-queries","_version":1,"created":true}
Loading visualization Top-Thrift-RPC-calls-with-errors:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-Thrift-RPC-calls-with-errors","_version":1,"created":true}
Loading visualization Top-Thrift-RPC-methods:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Top-Thrift-RPC-methods","_version":1,"created":true}
Loading visualization Total-number-of-HTTP-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Total-number-of-HTTP-transactions","_version":1,"created":true}
Loading visualization Total-time-spent-in-each-MongoDB-collection:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Total-time-spent-in-each-MongoDB-collection","_version":1,"created":true}
Loading visualization Web-transactions:
{"_index":"topbeat-03-30-2016","_type":"visualization","_id":"Web-transactions","_version":1,"created":true}
Loading dashboard HTTP:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"HTTP","_version":1,"created":true}
Loading dashboard MongoDB-performance:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"MongoDB-performance","_version":1,"created":true}
Loading dashboard MySQL-performance:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"MySQL-performance","_version":1,"created":true}
Loading dashboard Packetbeat-Dashboard:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"Packetbeat-Dashboard","_version":1,"created":true}
Loading dashboard PgSQL-performance:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"PgSQL-performance","_version":1,"created":true}
Loading dashboard Thrift-performance:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"Thrift-performance","_version":1,"created":true}
Loading dashboard Topbeat-Dashboard:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"Topbeat-Dashboard","_version":1,"created":true}
Loading dashboard Winlogbeat-Dashboard:
{"_index":"topbeat-03-30-2016","_type":"dashboard","_id":"Winlogbeat-Dashboard","_version":1,"created":true}
Loading index pattern filebeat-*:
{"_index":"topbeat-03-30-2016","_type":"index-pattern","_id":"filebeat-*","_version":1,"created":true}
Loading index pattern packetbeat-*:
{"_index":"topbeat-03-30-2016","_type":"index-pattern","_id":"packetbeat-*","_version":1,"created":true}
Loading index pattern topbeat-*:
{"_index":"topbeat-03-30-2016","_type":"index-pattern","_id":"topbeat-*","_version":1,"created":true}
Loading index pattern winlogbeat-*:
{"_index":"topbeat-03-30-2016","_type":"index-pattern","_id":"winlogbeat-*","_version":1,"created":true}
Then I ran logstash:
output {
stdout{ codec => rubydebug}
amazon_es {
hosts => ["search-123.us-west-2.es.amazonaws.com"]
region => "us-west-2"
aws_access_key_id => 'ABC'
aws_secret_access_key => 'XYZ'
index => "topbeat-03-30-2016"
}
}
Still when I go to kibana, I cannot see the beats dashboards :(
And yes I followed this guide:
https://www.elastic.co/guide/en/beats/libbeat/current/view-kibana-dashboards.html
One thing to note, is I never see the predefined index patterns it supposedly creates from the guide above.
alexanderfichel commented
From the AWS console the dashboards actually do get created, but nothing seen in actually kibana dashboard
monicasarbu commented
@Gittyhup By default, the load.sh script is loading the Kibana dashboards to the .kibana index. If you want to use .kibana4 instead then you can run the following command:
./load.sh -url "search-123.us-west-2.es.amazonaws.com" -index .kibana4
The topbeat-* index pattern is used by Topbeat to store the system statistics.
alexanderfichel commented
@monicasarbu Thank you!!! I had to change to .kibana-4 instead though. But it finally works!