ELK stack can't read lumberjack unencrypted data bag that it created

[martinb3]

Reported by @chri7765. If elkstack has no lumberjack keypair, it will generate one and store it in an un-encrypted data bag. On the next agent to converge, if no other encrypted data bags exist, it will pick up the key it generated no problem. But it will fail if any other encrypted data bags exist.

Error:

  Recipe Compile Error in /var/chef/cache/cookbooks/wrapper/recipes/default.rb
  ================================================================================

  Chef::EncryptedDataBagItem::DecryptionFailure
  ---------------------------------------------
  Error decrypting data bag value: 'wrong final block length'. Most likely the provided key is incorrect

  Cookbook Trace:
  ---------------
    /var/chef/cache/cookbooks/elkstack/recipes/_secrets.rb:77:in `from_file'
    /var/chef/cache/cookbooks/elkstack/recipes/agent.rb:49:in `from_file'
    /var/chef/cache/cookbooks/wrapper/recipes/elk_agent.rb:11:in `from_file'
    /var/chef/cache/cookbooks/wrapper/recipes/default.rb:14:in `block in from_file'
    /var/chef/cache/cookbooks/wrapper/recipes/default.rb:10:in `each'
    /var/chef/cache/cookbooks/wrapper/recipes/default.rb:10:in `from_file'

  Relevant File Content:
  ----------------------
  /var/chef/cache/cookbooks/elkstack/recipes/_secrets.rb:

   70:    lumberjack_secrets = Chef::DataBagItem.new
   71:    lumberjack_secrets.data_bag(lumberjack_data_bag)
   72:    lumberjack_secrets.raw_data = secrets
   73:    lumberjack_secrets.save
   74:  end
   75:  
   76:  # now try to use the data bag
   77>> if !lumberjack_secrets.nil? && lumberjack_secrets['key'] && lumberjack_secrets['certificate']
   78:    node.run_state['lumberjack_decoded_key'] = Base64.decode64(lumberjack_secrets['key'])
   79:    node.run_state['lumberjack_decoded_certificate'] = Base64.decode64(lumberjack_secrets['certificate'])
   80:  elsif !lumberjack_secrets.nil?
   81:    fail 'Found a data bag for lumberjack secrets, but it was missing \'key\' and \'certificate\' data bag items'
   82:  elsif lumberjack_secrets.nil?
   83:    fail 'Could not find an encrypted or unencrypted data bag to use as a lumberjack keypair, and could not generate a keypair either'
   84:  else
   85:    fail 'Unable to complete lumberjack keypair configuration'
   86:  end


  Running handlers:
[2014-11-13T08:35:59-07:00] ERROR: Running exception handlers
  Running handlers complete
[2014-11-13T08:35:59-07:00] ERROR: Exception handlers complete
[2014-11-13T08:35:59-07:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
  Chef Client failed. 1 resources updated in 16.827280415 seconds
[2014-11-13T08:36:00-07:00] ERROR: Error decrypting data bag value: 'wrong final block length'. Most likely the provided key is incorrect
[2014-11-13T08:36:00-07:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)