danieldeleo/beam-workshop

Getting Started with an Apache Beam Development Environment

Getting Started with an Apache Beam Development Environment

Link to Slides

Install Cloud SDK

• (Optional) Request a Test Google Account
• Verify if already installed by typing gcloud in bash terminal
• Follow the appropriate quickstart for your OS
• Initialize tool
  • gcloud init
    • Create your own project when prompted
• Set default application credentials
  • gcloud auth application-default login

Download IDE (Integrated Development Environment)

For Java Developers:

• Download/Install IntelliJ Community Edition
• Download/Install OpenJDK 8 or Java JDK 8

For Python Developers:

• Download/Install PyCharm Community Edition

For Developers with only a browser:

• Clone GitHub repos in cloud shell and use code editor

  • Shorcut to automatically clone DataflowTemplates

    • Alternatively in cloud shell run:

    git clone https://github.com/GoogleCloudPlatform/DataflowTemplates.git

  • Shortcut to automatically clone beam-workshop

    • Alternatively in cloud shell run:

    git clone https://github.com/danieldeleo/beam-workshop.git

Launch your IDE

On the welcome screen:

• Select Checkout from Version Control
• Choose Git from the dropdown

Enter git repo URL:

• For Java Developers enter:
  • https://github.com/GoogleCloudPlatform/DataflowTemplates.git
• For Python Developers enter:
  • https://github.com/danieldeleo/beam-workshop.git

Running Beam in Cloud Dataflow

Set the following required pipeline arguments:

• --runner=DataflowRunner
• --project=YOUR_PROJECT_ID

More Dataflow Pipeline Options

Build and Run Beam SDK Examples

Launch IntelliJ and on the welcome screen:

• Select Checkout from Version Control
• Choose Git from the dropdown
• Enter git repo URL: https://github.com/apache/beam.git

Python

The following commands should be run in the sdks/python directory

• virtualenv env
• source env/bin/activate
• pip install -e .[gcp]