Dynamic Inventory with PuppetDB Fails with Self-Signed Certificate

[sjusko]

Issue Overview

When attempting to use Puppet Bolt with dynamic inventory using PuppetDB as the inventory source, the operation fails when PuppetDB is configured with a self-signed certificate.

Even though the certificate and key are specified in the bolt-project.yaml configuration file, and the CA certificate has been added to the system-wide trust store, the connection to PuppetDB fails with certificate verify failed.

Steps to Reproduce

1. Configure Puppet Bolt in bolt-project.yaml and inventory.yaml with the following content:

   ---
   name: bolt_poc_project
   modules: []
   puppetdb:
     server_urls:
       - 'https://<puppetdb_url>:443'
     cacert: './certs/ca.crt'
     cert: './certs/client.crt'
     key: './certs/client.key'

   ---
   groups:
   - name: test
      targets:
         - _plugin: puppetdb
         query: "inventory[certname] { facts.fqdn ~ '.*' }"
   config:
   ssh:
      host-key-check: false

   Ensure that the paths specified for cacert, cert, and key are correct.

2. Add the CA certificate ca.crt to the system-wide trust store using the following command:

   Arch Linux:

   cp ./certs/ca.crt /etc/ca-certificates/trust-source/anchors/puppetdb_cacert.crt
   update-ca-trust

   Ubuntu:

   cp ./certs/ca.crt /usr/local/share/ca-certificates/puppetdb_cacert.crt
   update-ca-certificates

3. Attempt to use Puppet Bolt with PuppetDB as the inventory source.

   ❯ bolt inventory show
   Request to puppetdb at https://<puppetdb_url>:443 failed with Failed to query PuppetDB: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate).
   Error executing plugin puppetdb from resolve_reference in puppetdb: Failed to connect to all PuppetDB server_urls: https://<puppetdb_url>:443.

Expected Behavior

Puppet Bolt should successfully connect to PuppetDB and retrieve the dynamic inventory information.

Actual Behavior

Puppet Bolt fails to connect to PuppetDB with certificate error:

Request to puppetdb at https://<puppetdb_url>:443 failed with Failed to query PuppetDB: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate).
Error executing plugin puppetdb from resolve_reference in puppetdb: Failed to connect to all PuppetDB server_urls: https://<puppetdb_url>:443.

Environment

• Puppet Bolt Version: 3.22.0
• PuppetDB Version: puppet/puppetdb:7.8.0
• Operating System and Version: Arch Linux / Ubuntu 18.04

Additional Information

• A curl request to the PuppetDB API using the same client and CA certificates works correctly, indicating that the system-wide trust store is properly configured.

  curl -v -X GET https://<puppetdb_url>/pdb/query/v4 \
      --data-urlencode 'query=nodes[certname] {}' \
      --cert ./certs/client.crt \
      --key ./certs/client.key | jq -r '.[].certname'

• Bolt with PuppetDB inventory works fine when I add http.ssl_config.set_default_paths to my local puppetdb/client.rb (see below).

  ❯ grep set_default_paths /opt/puppetlabs/bolt/lib/ruby/gems/2.7.0/gems/bolt-3.22.0/lib/bolt/puppetdb/client.rb
        @http.ssl_config.set_default_paths