/swift-on-cloud

Primary LanguageC

Swift cloud on Google Compute Engine

Sign up online

  1. Go to Google’s compute engine page

  2. Create a new project and fill in the project and project id.

  3. Select the project and then select compute engine from the left panel.

  4. Enter billing information.

Note
 You should sign into your google account. Once the billing info is set, you might need to wait a few minutes. Please note the project-name and project-id in the config file in the swift-on-cloud/compute-engine folder.

Setup local-machine

On your local-machine setup Google’s cloud SDK based on the document here:

curl https://dl.google.com/dl/cloudsdk/release/install_google_cloud_sdk.bash | bash
# Restart bash session for the changes to kick in.

Authenticate via browser, by clicking the link and pasting the response to the commandline

gcloud auth login

Get the swift-on-cloud repository from git

git clone git@github.com:yadudoc/swift-on-cloud.git
cd swift-on-cloud/compute-engine

Manage you Cloud resources

Update the file swift-on-cloud/compute-engine/configs in the cloned repository from the previous step, with the following information:

  • GCE_PROJECT : Name of project selected during the compute-engine sign up.

  • GCE_PROJECTID : This is the unique ID for your project selected during compute-engine sign up.

  • WORKER_MACHINE_TYPE, HEADNODE_MACHINE_TYPE: VM types for workers and headnode, choose between:

    • 'f1-micro'

    • 'g1-small'

    • 'n1-standard-<1,2,4,8>'

    • 'n1-<highmem/highcpu>-<2,4,8>'

  • GCE_WORKER_COUNT : Number of workers to start at setup (pick a number between 1 and 20)

  • GCE_ZONE : GCE Region to use (Multiple regions are not supported now). Choose between:

    • 'us-central1-a' and 'us-central1-b' (with Sandy bridge processors)

    • 'europe-west1-a' and 'europe-west1-b' (with Sandy bridge processors)

    • 'asia-east1-a' and 'asia-east1-b' (with Ivy bridge processors)

Note
 Read more about Compute-engine zones here
Note
 GCE_WORKER_COUNT directly affects the cost. If you require more than 22 nodes including the headnode, file a request to increase your resource quotas.
Warning
 Do NOT change the images for the worker and headnode to point at images which have not been explicitly setup with swift.

Start your cloud setup!

Once you finish editing the configs file with your preferences, start the cloud instances by sourcing the setup script.

The setup script will setup firewall rules, copy over the required images and start a headnode and the requested number of worker instances.

# Must source the setup script.
source setup.sh
Note
 The setup script will ask for a passphrase when you run it for the first time. If you give a passphrase, it will be required when you attempt to connect to the cloud instances.
Note
 The setup.sh script may not work on shells besides bash.

Run swift from your local machine

Now, once you’ve configured and started you cloud resources we can move on to trying the swift-cloud-tutorial. To run the cloud tutorial from your local machine :

cd swift-on-cloud/swift-cloud-tutorial
source setup.sh
Note
 You must source the setup.sh script.

Run the tutorial on the cloud

Once your cloud resources have been configured and setup, you can run the swift-cloud-tutorial directly from the cloud. The cloud resources created include a headnode, to which you would connect to, and the several nodes in worker roles which would do computations in parallel.

To run the tutorial, first connect to the headnode:

# Connect to the Headnode
connect headnode
# This will have you logged in to the headnode on the cloud

Miscellanious operations supported:

# Connect to the Headnode
connect headnode

To see resources use :

list_resources

To ssh to any resource listed :

# Specify the resource name as listed by list_resources here
# If resource_name is omitted, connect will try to connect to the
# headnode
connect <resource_name>

To stop all resources use from your local machine. Please wait for a couple of minutes for the command to finish. Confirm that all resources have been removed using list_resources. :

# This will delete the headnode as well as all workers.
# This command will take a few minutes to execute
dissolve
# Use list_resources to check if any resources still linger
list_resource

To add more worker nodes use:

# The number of nodes you can create is limited by the quota's set by google.
# To increase quotas contact google using the change request form available
# under Your project / Compute engine / Quotas tab in developer console
start_n_more <Number of nodes>

# Alternatively update the configs with the total number of nodes you require
# and rerun the setup script
source setup.sh

Note: Creating an Image Link