Docker Compose Stack with Grafana Loki, Promtail and the Grafana Explore Logs plugin.
Note
For a full example with prometheus, alertmanager, cadvisor, node-exporter and loki, please see the following repo:
Start the stack with:
make up
Access nginx on http://localhost:8080
Then navigate to grafana on http://localhost:3000 and select explore on the left and select the container:
And you will see the logs:
The grafana explore-logs app can be found under explore:
Any application that logs to Loki can be discovered here:
![image](https://private-user-images.githubusercontent.com/567298/348416962-2d03c559-1046-4897-9429-eb9bc6fc89ee.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjk3ODgzNDMsIm5iZiI6MTcyOTc4ODA0MywicGF0aCI6Ii81NjcyOTgvMzQ4NDE2OTYyLTJkMDNjNTU5LTEwNDYtNDg5Ny05NDI5LWViOWJjNmZjODllZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMDI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTAyNFQxNjQwNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iMjViNTc3OTBlNDhhODA0OGZiYzQyNTk2YmQ2MGEzYTZlYzRlOTE3NjQ1NjQ0Y2JlMWRkMWRkZWQxNmNkZDJmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.Z67C-XbAVDw2jQh0dIlmJ_JYvYbwVENa0Deoshaxjsk)
Here we can view the log content for the selected service_name
:
![image](https://private-user-images.githubusercontent.com/567298/348417321-1eb0e32f-5019-437e-b760-b1a63cbc8beb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjk3ODgzNDMsIm5iZiI6MTcyOTc4ODA0MywicGF0aCI6Ii81NjcyOTgvMzQ4NDE3MzIxLTFlYjBlMzJmLTUwMTktNDM3ZS1iNzYwLWIxYTYzY2JjOGJlYi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMDI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTAyNFQxNjQwNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kMGQ3NTIzOTlhZWE4M2QyZjdkNzM5NTY0Y2FkNTUzMzYzODc0ZDk2ZTljNjkzZWJjOTg4MGY0NGY1NzFhMmJiJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.6eDYcSrupDlQm2gnGA6gkRjS1dJ9Dh5tGWFV-MaFBoo)
This view we can explore the different labels thats available for our selected service:
![image](https://private-user-images.githubusercontent.com/567298/348417638-a222ab21-5c3f-4f66-b01e-9e882c1a8df2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjk3ODgzNDMsIm5iZiI6MTcyOTc4ODA0MywicGF0aCI6Ii81NjcyOTgvMzQ4NDE3NjM4LWEyMjJhYjIxLTVjM2YtNGY2Ni1iMDFlLTllODgyYzFhOGRmMi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMDI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTAyNFQxNjQwNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02NGNlNmJlYTYyMTFlMjU1ZjcyYWY5OGY5OWZhN2IxZmQ0YTFiM2FmYzA0NzJkNzY2YmU5Mjg4MGIwMzFlNGUwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.5PoZy2FqVkKDp4HtLQbIN452C4tf-r8tR_w7BH1nN5c)
For example, we can select the level
label, which will then append that to our LogQL query and present us with this view:
![image](https://private-user-images.githubusercontent.com/567298/348417992-f805b460-6f70-4298-a8b3-2118e4931266.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjk3ODgzNDMsIm5iZiI6MTcyOTc4ODA0MywicGF0aCI6Ii81NjcyOTgvMzQ4NDE3OTkyLWY4MDViNDYwLTZmNzAtNDI5OC1hOGIzLTIxMThlNDkzMTI2Ni5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMDI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTAyNFQxNjQwNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT03ZjEyMTAyM2EzNTk4Mzk2ODU2MDliMjg3YmQxYWRjYzljZjVkNzJiNDQxNzY5ZWYwNjZjNjFjNTg5YmNhNDc2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.qAMd40ESTWkf3E6eIoWcjbfWfGvss3BjSpKVWVNqJ34)
We can then include error
for example and we will see all the error logs for our selected service:
![image](https://private-user-images.githubusercontent.com/567298/348418134-d1502c06-5cae-498d-81e0-470afd7890dd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjk3ODgzNDMsIm5iZiI6MTcyOTc4ODA0MywicGF0aCI6Ii81NjcyOTgvMzQ4NDE4MTM0LWQxNTAyYzA2LTVjYWUtNDk4ZC04MWUwLTQ3MGFmZDc4OTBkZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMDI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTAyNFQxNjQwNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04N2VmMzA0NjNiMWIxMTNkODg3OTdjMGQ4MmUwYzg1N2RlNjk1MDNjYzFhNmNjZjg2ODlkMjg2NWRlYWU3NjEwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.-ahKFFsW61wXzEBT8mFc1F1u9fzTJtQgh7PfbmXzy8U)
We can then also select "Open in Explore" and our query will be pre-populated for us, eg. {service_name="tempo-ingester", level="error"} | logfmt
:
![image](https://private-user-images.githubusercontent.com/567298/348418833-974b8916-c9f6-466a-994e-52d10e771353.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjk3ODgzNDMsIm5iZiI6MTcyOTc4ODA0MywicGF0aCI6Ii81NjcyOTgvMzQ4NDE4ODMzLTk3NGI4OTE2LWM5ZjYtNDY2YS05OTRlLTUyZDEwZTc3MTM1My5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMDI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTAyNFQxNjQwNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xODMzYzhiNWFlNTgyMmRhMTA1OWQwY2QxYzY0NGQ0ZTIzNDBlMDg5ZGFkNmFhY2ZlZDI0NGZiYTIyYzBkNjExJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9._U2cpUM1zDTdSSn17G06Jdd-vMzfCs1E6iTttTGfgQQ)
We can also explore logs by fields:
![image](https://private-user-images.githubusercontent.com/567298/348419574-3b459123-f132-422c-8ac3-ea69898e7f94.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjk3ODgzNDMsIm5iZiI6MTcyOTc4ODA0MywicGF0aCI6Ii81NjcyOTgvMzQ4NDE5NTc0LTNiNDU5MTIzLWYxMzItNDIyYy04YWMzLWVhNjk4OThlN2Y5NC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMDI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTAyNFQxNjQwNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yNDAzZDczNWIzY2EwM2IzNTBlODZjZTE4Y2M3NmRlOTEzOWQ4MzI3ZDYzOWM3NDFmN2NhMjZiNTE2YTU3NjdlJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.k1BeAuwTuqcy5yb-uU8UESJaZzuXVQYcx0OD22I1IGk)
As well as patterns:
![image](https://private-user-images.githubusercontent.com/567298/348419732-f2adc9b8-b808-45a5-8dab-d17accd31aa0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjk3ODgzNDMsIm5iZiI6MTcyOTc4ODA0MywicGF0aCI6Ii81NjcyOTgvMzQ4NDE5NzMyLWYyYWRjOWI4LWI4MDgtNDVhNS04ZGFiLWQxN2FjY2QzMWFhMC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMDI0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTAyNFQxNjQwNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05ODhmMzQxMGJjM2FkYjYxNWFlNmY2NTcyZDIyOTRhZGQ4MzA1YmIzNGVmOTQwMzVmNDViNDU2OWVkYjYzOTczJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.VXTTCuZGBHa1BOUUbFS9ashgvxDitw4ctyZCCt48P-M)
For more information about exploring logs without LogQL, view Grafana's blog post: