404 Not Found if default environment isn't populated in PuppetDB
Closed this issue · 5 comments
I have been thinking of migrating my existing Puppet/PuppetDB/Puppetboard setup that is currently in a virtual machine and switching it to docker components instead to make it easier to maintain/upgrade. I found the Puppet-In-Docker example https://github.com/puppetlabs/puppet-in-docker-examples/blob/master/compose/docker-compose.yml and it seems to run as expected, but as you have noticed the puppetboard component hasn't been maintained so I went looking around and found yours that appears to be a drop in replacement that is current (as yours is using the same port 8000 and from what I can see it is expecting the same configuration as the original one from puppet labs but still being built). So when I take the puppetboard section of that docker-compose.yml file and change it from puppet/puppetboard to bootc/puppetboard it looks like it should work, but I get a steady stream of the following messages in the logs:
puppetboard_1 | ERROR:pypuppetdb.api:Could not reach PuppetDB on puppetdb:8080 over HTTP.
puppetboard_1 | ERROR:puppetboard.utils:HTTPConnectionPool(host='puppetdb', port=8080): Max retries exceeded with url: /pdb/query/v4/environments (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9eff212070>: Failed to establish a new connection: [Errno 111] Connection refused'))
puppetboard_1 | 127.0.0.1 - - [22/Apr/2020:00:04:00 +0000] "GET / HTTP/1.1" 500 290 "-" "curl/7.67.0"
Any idea why the connection to puppetdb would be failing - when I am writing this the puppetdb image from puppet/puppetdb was updated today - so I dont think it is a case of puppetdb being too old for the puppetboard to talk to it.
The puppetexplorer included in that docker-compose is able to talk to the puppetdb, so it is not a case that the puppetdb isnt working.
I am curious if you can provide any insight?
Well, for anyone else in this situation, I have gotten closer (but no fix yet) - I noticed that the puppetdb container was dying and I found this https://tickets.puppetlabs.com/browse/PDB-4540 - you have to update the ssl volume mounts for newer puppet releases as they moved the ssl file locations. This fixes the puppetdb dying, but I am still not able to get puppetboard to see the puppetdb.
puppetboard_1 | 127.0.0.1 - - [22/Apr/2020:04:12:14 +0000] "GET / HTTP/1.1" 404 232 "-" "curl/7.67.0"
attaching into the puppetdb and puppetboard containers I can see that the names are resolving properly, and looking at the open ports the 8080 port is opened for 0.0.0.0 access and not localhost so I think it should be reachable, but that is the limit of my ability to debug this.
Hmm, well I have figured out how to get to the puppetdb certs, and I have verified that I can ping the puppetdb instance from the puppetboard instance, but even with using the HTTPS connection I get the same results of connection refused -
/opt/puppetdbssl/private_keys # ping puppetdb
PING puppetdb (192.168.112.5): 56 data bytes
64 bytes from 192.168.112.5: seq=0 ttl=64 time=0.091 ms
64 bytes from 192.168.112.5: seq=1 ttl=64 time=0.096 ms
puppetboard_1 | ERROR:pypuppetdb.api:Could not reach PuppetDB on puppetdb:8081 over HTTPS.
puppetboard_1 | ERROR:puppetboard.utils:HTTPSConnectionPool(host='puppetdb', port=8081): Max retries exceeded with url: /pdb/query/v4/environments (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fc7ce443760>: Failed to establish a new connection: [Errno 111] Connection refused'))
puppetboard_1 | 127.0.0.1 - - [22/Apr/2020:12:43:36 +0000] "GET / HTTP/1.1" 500 290 "-" "curl/7.67.0"
I am not a python developer, so I am having to learn my way through this logging info - I am now noticing that the cannot connect messages seem to end after I see a log message from the puppetdb that jetty has started, but I still cant get the puppetboard to load. After jetty has started in the heartbeat/health section of the log I keep seeing 404s from puppetboard
puppetdb_1 | 127.0.0.1 - - [22/Apr/2020:12:43:58 +0000] "GET /status/v1/services/puppetdb-status HTTP/1.1" 200 247 "-" "curl/7.64.0"
puppetboard_1 | 127.0.0.1 - - [22/Apr/2020:12:43:58 +0000] "GET / HTTP/1.1" 404 232 "-" "curl/7.67.0"
puppetdb_1 | 192.168.112.4 - - [22/Apr/2020:12:43:58 +0000] "GET /pdb/query/v4/environments HTTP/1.1" 200 2 "-" "-"
puppet | 127.0.0.1 - - - 22/Apr/2020:12:44:00 +0000 "GET /status/v1/simple HTTP/1.1" 200 7 127.0.0.1 127.0.0.1 8140 12
so it looks like gunicorn is not running properly. I dont know if there is a retry limit on gunicorn that it is failing to start in time or if there is some other issue - looking through the gunicorn documentation I dont see anything about retry counts.
Any suggestions on where to look next ?
Well final update on this - figured it out :)
I went looking around in the puppetboard issues and I found this - voxpupuli/puppetboard#562 . Ends up being that puppetboard wont load unless you have a puppet configuration in use with an environment and a connected agent machine to start populating the puppetdb. Note that the puppetexplorer doesnt have this requirement which is why it was working.
Once I moved an existing puppet agent over to the new puppet master the puppetboard started to work.
workingish-docker-compose.yml.txt
I have attached a copy of my working docker-compose.yml file with the alterations I had to make to the one I posted in the original message on this. Since I already have another puppetmaster on my network, I called the puppet master I am running in this puppetdocker, so there are changes in the yml file to add that name as aliases on the certificates and I have assigned fixed ports on the host to expose the required ports from the various containers.
I hope this will help somebody else trying to get this working.
Sorry I haven't had a chance to look into this and reply! I'm really glad you figured it all out, and then posted your findings here for someone to pick up down the line. Thanks, and sorry I didn't come in to help earlier!
Well. one tweak - I just discovered today that I screwed up the puppet server ssl line - the ssl storage on the puppet server hasn't been moved under /opt and as a result if you restart the docker stack your certs are invalidated since they werent saved outside of the container. So the volumes section for Puppet in the compose file should look like
volumes:
- ./code:/etc/puppetlabs/code/
- ./puppet/ssl:/etc/puppetlabs/puppet/ssl/
- ./puppet/serverdata:/opt/puppetlabs/server/data/puppetserver/
(which is how it was in the original file)
and bootc - no problem with you not answering - you haven't had an issues posted here before so I wouldn't have been surprised if you weren't looking here for them. This gave me a chance to really learn how all of this stuff wires together while trying to debug it. And thank you for having a current version of the puppetboard available as a container - I had been using 0.21 of puppetboard before this. :)