/bosh-gcscli

GCS for BOSH

Primary LanguageGoApache License 2.0Apache-2.0

bosh-gcscli

GoDoc

A Golang CLI for uploading, fetching and deleting content to/from Google Cloud Storage. This tool exists to work with the bosh-cli and director.

This is not an official Google Product.

Installation

go get github.com/cloudfoundry/bosh-gcscli

Commands

Usage

bosh-gcscli --help

Upload an object

bosh-gcscli -c config.json put <path/to/file> <remote-blob>

Fetch an object

bosh-gcscli -c config.json get <remote-blob> <path/to/file>

Delete an object

bosh-gcscli -c config.json delete <remote-blob>

Check if an object exists

bosh-gcscli -c config.json exists <remote-blob>

Generate a signed url for an object

If there is an encryption key present in the config, then an additional header is sent

bosh-gcscli -c config.json sign <remote-blob> <http action> <expiry>

Where:

  • <http action> is GET, PUT, or DELETE
  • <expiry> is a duration string less than 7 days (e.g. "6h")

Configuration

The command line tool expects a JSON configuration file. Run bosh-gcscli --help for details.

Authentication Methods (credentials_source)

  • static: A service account key will be provided via the json_key field.
  • none: No credentials are provided. The client is reading from a public bucket.
  • <empty>: Application Default Credentials will be used if they exist (either through gcloud auth application-default login or a service account). If they don't exist the client will fall back to none behavior.

Running Integration Tests

  1. Ensure gcloud is installed and you have authenticated (gcloud auth login). These credentials will be used by the Makefile to create/destroy Google Cloud Storage buckets for testing.
  2. Set the Google Cloud project: gcloud config set project <your project>
  3. Generate a service account with the Storage Admin role for your project and set the contents as the environment variable GOOGLE_APPLICATION_CREDENTIALS, for example: 
    export project_id=$(gcloud config get-value project)

export service_account_name=bosh-gcscli-integration-tests
export service_account_email=${service_account_name}@${project_id}.iam.gserviceaccount.com
credentials_file=$(mktemp)

gcloud config set project ${project_id}
gcloud iam service-accounts create ${service_account_name} --display-name "Integration Test Access for bosh-gcscli"
gcloud iam service-accounts keys create ${credentials_file} --iam-account ${service_account_email}
gcloud project add-iam-policy-binding ${project_id} --member serviceAccount:${service_account_email} --role roles/storage.admin

export GOOGLE_SERVICE_ACCOUNT="$(cat ${credentials_file})"
export GOOGLE_APPLICATION_CREDENTIALS="$(cat ${credentials_file})"
export LC_ALL=C # fix `tr` complaining about "illegal byte sequence" on OSX
  4. Run the unit and fast integration tests: make test-fast-int
  5. Clean up buckets: make clean-gcs

Development

  • A Makefile is provided that automates integration testing. Try make help to get started.
  • gvt is used for vendoring.

Contributing

For details on how to contribute to this project - including filing bug reports and contributing code changes - please see CONTRIBUTING.md.

License

This tool is licensed under Apache 2.0. Full license text is available in LICENSE.