zeropingheroes/lancache-elk

debian upgraded kibana/elasticsearch and now my graphs are broken.

Closed this issue · 5 comments

billthecatt commented

Tried re-installing using install.sh but it appears the directions provided no longer work for the latest version of kibana/elasticsearch as provided with the latest debian stretch.

My versions are:
kibana (6.4.3) ...
elasticsearch (6.4.3) ...

Any ideas on how to get this working again? I really liked having data analytics around what was being downloaded from the cache during a LAN.

Thanks again for all these great projects.

billthecatt commented

How to rollback from kibana/elasticsearch 6.4.3 to 6.4.0 on debian stretch:

I learned how to do this here via googlefoo:
https://wiki.debian.org/RollbackUpdate

First remove the new kibana and elasticsearch:

apt-get remove kibana elasticsearch

Then, modify /etc/apt/preferences as follows:

Package: kibana
Pin: version 6.4.0
Pin-Priority: 1001

Package: elasticsearch
Pin: version 6.4.0
Pin-Priority: 1001

This will prevent apt-get update && apt-get upgrade from upgrading beyond what this project supports.

Then, re-install kibana and elastic search:

apt-get install kibana elasticsearch

<Notice that when this takes place 6.4.0 should downloaded & install>
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 207 MB of archives.
After this operation, 505 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 kibana amd64 6.4.0 [207 MB] <-- note 6.4.0 here.

The same should happen for elasticsearch:
Get:1 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 elasticsearch all 6.4.0 [91.4 MB] <-- Again, 6.4.0 here..

Then I ran the installer.sh and ran into this error:
{"message":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];","statusCode":403,"error":"Forbidden"}

If you get this error you may have had your disc full.. in which case kibana goes into read-only mode and this must be cleared using:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Appologies if this breaks some convention on how to run kibana/elasticsearch but I really wanted my analytics back on my cache sooooo.. This is what I did to make it happen, your experience may vary, hope this helps someone else.

ilumos commented

What error(s) were you getting, if any?

I'll see if I can reproduce them, and/or update this project for the latest version of ELK.

Thanks!

billthecatt commented

No specific error messages, It was just that once you upgrade to the latest kibana/elasticsearch the dashboards aren't there anymore and the installer/ install docs doesn't seem to work on the new version.

billthecatt commented

It seems that the latest version of logstash debian is shipping has been playing havoc with the older versions. As a result I've had to roll back to logstash v 1:6.4.0-1.

It turns out this was harder to do than I thought it would be and required me restarting from scratch.. Since this project doesn't have an uninstaller.sh.. I'll provide the one I came up with:

#Kill all bogon/zombie services
killall nginx
killall kibana
killall logstash
killall elasticsearch
killall java

#Stop the services
/bin/systemctl start elasticsearch
/bin/systemctl start logstash
/bin/systemctl start kibana
/bin/systemctl start nginx

#Load the new service file
/bin/systemctl daemon-reload

#Set services to not start at boot
/bin/systemctl disable elasticsearch
/bin/systemctl disable logstash
/bin/systemctl disable kibana
/bin/systemctl disable nginx

#remove all the things
apt-get autoremove logstash kibana elasticsearch nginx -y
apt-get --purge remove logstash kibana elasticsearch nginx -y
apt-get autoremove -y && sudo apt-get autoclean -y

#remove even harder to clean up any remenants
find / | grep logstash | sudo xargs rm -rf
find / | grep kibana | sudo xargs rm -rf
find / | grep elasticsearch | sudo xargs rm -rf
find / | grep nginx | sudo xargs rm -rf

#Kill all bogon/zombie services one more time just for good measure
killall nginx
killall kibana
killall logstash
killall elasticsearch
killall java

Once that's completed your systems should be clear of any files that block the installer.sh from doing the right thing. You can then do:

apt-get update
apt-get apt-get install elasticsearch=6.4.0
apt-get apt-get install logstash=1:6.4.0-1
apt-get apt-get install kibana=6.4.0

and run installer.sh without further issues.

Hope this helps someone else having fun with this project on deb 9.6.

ilumos commented

Thanks for this.

I've updated to Ubuntu 18.04 and changed to a different version of Java which should solve.