Enhancement: Remote File Provisioner (s3)

[estenrye]

Enhancement: Remote File Provisioner (s3)

Use Case
As a user, I need the capability to provision one or more files from a private S3 bucket to an image without committing my access_key and secret_key values to the repository. Ideally, packer would pull these values similarly to how the Amazon EC2 builder handles this by retrieving them from environment variables or the .aws folder in the user's profile directory.

Example AWS Profile Config

[default]
output = json
region = us-east-2

[profile wasabi]
region = us-east-1
output = json
s3 =
    endpoint_url = https://s3.wasabisys.com
s3api =
    endpoint_url = https://s3.wasabisys.com

[plugins]
endpoint = awscli_plugin_endpoint

Inputs

• objects
  • This parameter would provide the provisioner with the list of S3 objects to retrieve from the S3 api.
  • This parameter would be required.
  • This parameter would be a list of objects with the following data structure to pull a single file:

    { 
        "src":"s3://vagrant-cloud/lab-dc/hyperv-1.0.20190518.205645.box",
        "target:":"/path/to/target/location.box"
    }

  • If the user specifies a folder in the target property, the file should be copied to the target directory and its filename should be preserved.
  • If the user specifies an S3 folder in the src property, all files and folders under that path should be recursively to the target directory and their names and directory structure should be preserved.
  • The target property must support Windows paths using forward slashes. Example: C:/tfs and c:/tfs both are valid.
• endpoint
  • This parameter would provide the provisioner with the S3 api endpoint that the file(s) reside on.
  • This parameter would be optional and would default to:
    • The s3:endpoint_url value in the AWS Profile configuration (if specified) when profile or the AWS_PROFILE environment variable is defined.
    • The AWS API endpoint.
  • This parameter provides interoperability with other vendors that support the S3 api spec.
• region
  • This parameter specifies the region in which the bucket is stored.
  • This parameter would be optional and would default to:
    • The value defined in the AWS profile configuration file (if specified) when profile or the AWS_PROFILE environment variable is set.
    • The value defined in the AWS_DEFAULT_REGION environment variable (if specified)
    • us-east-1 if none of the above are specified.
• profile
  • This parameter would provide the provisioner the name of the AWS profile configuration to use when communicating to the S3 api.
  • This parameter would be optional and would default to the AWS_PROFILE environment variable.
• access_key
  • This parameter would provide the provisioner with the access key required to authenticate to the S3 api endpoint.
  • This parameter would be optional and would default to:
    • The value defined in the AWS profile configuration if profile or the AWS_PROFILE environment variable are specified.
    • The value defined in the AWS_ACCESS_KEY_ID environment variable (if specified)
• secret_key
  • This parameter would provide the provisioner with the secret key required to authenticate to the S3 api endpoint.
  • This parameter would be optional and would default to:
    • The value defined in the AWS profile configuration if profile or the AWS_PROFILE environment variable are specified.
    • The value defined in the AWS_SECRET_ACCESS_KEY environment variable (if specified)

Example Configurations

Folder-to-Folder Configuration

{
  "provisioners": [
    {
      "type": "s3",
      "profile": "wasabi",
      "objects": [
        {
            "src":"s3://vagrant-cloud/domain-certificates",
            "target":"/opt/ldap-certs"
        }
      ]
    }
  ]
}

File-to-Folder Configuration

{
  "provisioners": [
    {
      "type": "s3",
      "endpoint":"https://s3.wasabisys.com",
      "region":"us-east-2",
      "access_key":"my-access-key",
      "secret_key":"my-secret-key",
      "objects":[
        {
            "src":"s3://vagrant-cloud/domain-certificates/domain.crt",
            "target":"/opt/ldap-certs"
        }
      ]
    }
  ]
}

File-to-File Configuration

{
  "provisioners": [
    {
      "type": "s3",
      "profile": "wasabi",
      "endpoint":"https://s3.wasabisys.com",
      "region":"us-east-2",
      "objects":[
        {
            "src":"s3://vagrant-cloud/domain-certificates/domain.crt",
            "target":"C:/opt/ldap-certs/domain.crt"
        }
      ]
    }
  ]
}

Multi-Bucket Configuration

{
  "provisioners": [
    {
      "type": "s3",
      "access_key":"my-access-key",
      "secret_key":"my-secret-key",
      "objects":[
        {
            "src":"s3://vagrant-cloud/domain-certificates/domain.crt",
            "target":"/opt/ldap-certs/domain.crt"
        },
        {
            "src":"s3://other-bucket/ssh-keys/user",
            "target":"/home/user/.ssh"
        }
      ]
    }
  ]
}

[rickard-von-essen]

So for Amazon builders this is fairly trivial using instance roles, so I guess this is mostly for copying files to other builders?

[rickard-von-essen]

Also this is fairly simple to do with a chain of shell-local and file provisioner but then it will rely more on having the appropriate tools installed locally.

[estenrye]

Yes, this is largely a post processor for the other builders. I don't have access to an instance role in a hyperv or virtualbox builder scenario, but I may have data that is too sensitive to store in a source control repository but still needs to be consistent for everyone committing to the repo.

Right now I am managing it in scripts that copy the files locally that have to be run before I do the build process. Not the end of the world mind you, but requires an additional amount of environment setup that would be nice to eliminate if this feature existed.

Does the file provisioner have the capability to pull remote files via an http or https uri?

[estenrye]

I am also capturing it here because I think it would be a fun feature to work on if I ever get the time to get around to working on it. Figured by posting it here I can get some discussion and feedback on the idea before I run too far down a path.

[estenrye]

I didn't think about the instance role use case though, my AWS is a bit rusty would it be easy for a provisioner like this to grab the instance role based on the builder type?

[rickard-von-essen]

So an EC2 that has an instance role assigned to it get temporary credentials that it can use to access the AWS API based on the permissions allowed by the instance role's attached policies. See IAM Roles for Amazon EC2.

[rickard-von-essen]

Currently the file provisioner only handles local files. There has been talk about creating a new provisioner using the go-getter, but no work have been done or planed for this to my knowledge.

[estenrye]

That was the other enhancement I was thinking about writing up this weekend.

I had two use cases in mind when I thought this up:

• For private S3 object stores, this functionality is needed to retrieve the data.
• For public S3 object stores, a simple get would be enough.

[azr]

Hello here, like Rickard said, I think this could be a go-getter provisioner, similar to the file provisioner but that uses the go-getter internally. The go-getter understands s3 urls (and many others) and will try to use the system's credentials to get files.

S3 takes various access configurations in the URL. Note that it will also read these from standard AWS environment variables if they're set.

[azr]

Also if you would like to give it a try @estenrye, I would be super happy to review. 🙂

[SwampDragons]

I think this is a cool idea, but we're freezing further plugin contributions for now. That said, this could be a good candidate for a community provisioner, and if someone ends up writing it we'd be happy to add to the website. But since this isn't work I think we'll be doing in the main repo, at least in the near future, I'm going to close.

[Deleted user]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.