sbayd/filebeat-caprover

Filebeat config for nginx and other logs in caprover docker containers

filebeat-caprover

Filebeat config for Nginx and other logs in Caprover Docker containers. This guide helps you to setup Filebeat to collect nginx and other logs on your Caprover instance then sends them to your elastic / kibana instances.

I had diffuculties and spend a few hours while integrating filebeat to my caprover instance, so I hope this helps someone!

Filebeat helps you to read log files and send them to Elastic search & Kibana instances on your servers. Useful for monitoring your apps on Caprover.

This guide assumes you already have;

• Running Caprover instance
• ElasticSearch app on your Caprover
• Kibana app on your Caprover

If any of the above are missing, this guide is not for you.

Lets begin!

1. Open your Caprover panel and go to "Apps"
2. On the create a new app panel, write a name for filebeat instance
3. Make sure you've selected the "Has Persistent Data" option

Should look like this:

4. Click to newly created app and go to app settings.
5. Check "Do not expose as web-app" for security, we'll not use it from outside.

Should look like this:

6. Go to App Configs page
7. Add following environment variables for your Elastic search and Kibana instances. (You can find the HTTP URLS from your apps main page.)

key	value	example
KIBANA_HOST	your kibana host	http://srv-captain--kibana
ELASTIC_HOST	your elastic host with internal port	http://srv-captain--elastic:9200

Should look like this:

8. Click add persistent data/directory button, and click to "Set specific host" option.

We'll open some docker directories to let our docker container to read other container logs from our docker host server.

Add following values

Path in App	Path in Host
/var/lib/docker/containers	/var/lib/docker/containers
/var/run/docker.sock	/var/run/docker.sock

Should look like this:

9. Go to pre deploy script and paste the following function. Make sure you have changed username to your host docker runner username. (root in my case). This script lets our docker container to access related logs files we specified in previous steps with correct credentials. Otherwise our docker container cannot access the logs even if we gave them as persistent directory. (You can also create a user group for this, we just need to give permissions somehow :))

var preDeployFunction = function (captainAppObj, dockerUpdateObject) {
    return Promise.resolve()
        .then(function(){
            dockerUpdateObject.TaskTemplate.ContainerSpec.User = 'yourusername';
            return dockerUpdateObject;
        });
};

Should look like this:

10. Click to Save & Update button on the bottom.
11. App Configuration is done! Now we need deployment.
12. You can use this repository for easy deployment, clone or fork this repo to your local computer.
13. If you want to change filebeat.yml file, make changes. Currently it is optimized for nginx configurations.
14. Go to the related folder in terminal
15. Use caprover deploy command (check the docs if you didn't make any deployment before)

You should see a similar output

16. It is ready! Go to your kibana public url and observe the logs!

17. Optional: close kibana/elastic nginx logs by adding acces_logs: off to nginx config of the kibana/elastic apps to prevent logging your actions on kibana to kibana :))