/heron-starter

Starter examples for Heron, the next generation streaming system from Twitter

Primary LanguageJavaOtherNOASSERTION

Heron Starter

Learn to use Heron!

Table of Contents

Getting started

Prerequisites

First, you will need a laptop that connect to internet using Wi-FI. In the laptop, make sure that you have java and git installed and in your user's PATH.

Second, you will need a Twitter account. If you do not have a Twitter account, please create one. From your Twitter account, you can follow the Heron experts:

@karthikz, @louis_fumaosong, @billgraham

Third, if you need to use Twitter firehose/Tweet stream for your idea, create a set of credentials by following the instructions in the section below.

Fourth (optional), partner with someone to form a group. We encourage 2/3 students per group.

Finally, make sure you have the heron-starter code available on your machine. Git/GitHub beginners may want to use the following command to download the latest heron-starter code and change to the new directory that contains the downloaded code.

$ git clone https://github.com/kramasamy/heron-starter.git && cd heron-starter

Twitter Credentials

  • Go to Twitter developer site and log in. Create a new account if you do not have one.

  • Click on the arrow on top right beside your account image and select "My applications".

  • Create a new application. If you already have one, you can use credentials from that application and skip this and the next step.

  • Fill in the details. You can use dummy values for all required fields.

  • In the application detail page note the Consumer key and Consumer secret fields under OAuth settings

  • Create a new access token at the bottom of the page. The Access token and Access token secret will be shown after a moment.

Overview

heron-starter contains a variety of examples of using Heron. If this is your first time working with Heron, check out these topologies first:

  1. ExclamationTopology: Basic topology written in all Java
  2. WordCountTopology: Basic topology for counting words all written in Java

After you have familiarized yourself with these topologies, take a look at the other topopologies in src/jvm/heron/starter/ such as RollingTopWords for more advanced implementations.

If you want to learn more about how Heron works, please head over to the Heron project page.

Using heron-starter with Maven

Install Maven

Maven is an alternative to Leiningen. Install Maven (preferably version 3.x) by following the Maven installation instructions.

Packaging heron-starter for use on a Aurora/Mesos cluster

You can package a jar suitable for submitting to a Aurora/Mesos cluster with the command:

$ mvn pom.xml package

This will package your code and all the non-Heron dependencies into a single "uberjar" at the path target/heron-starter-{version}-jar-with-dependencies.jar.

Submitting your jobs Locally

After compiling your package, you can submit your job locally by following the Quick Start Guide

$ heron submit local \
  ./target/heron-starter-{version}-jar-with-dependencies.jar \ # The path of the topology's jar file
  heron.starter.WordCountTopology \ # The topology's Java class
  WordCountTopology \ # The name of the topology
  --deploy-deactivated # Deploy in deactivated mode

This will submit the topology to your locally running Heron cluster but it won’t activate the topology. To activate, use the following Heron cli commands:

$ heron activate local WordCountTopology
$ heron deactivate local WordCountTopology
$ heron kill local WordCountTopology

Running unit tests

Use the following Maven command to run the unit tests that ship with heron-starter.

$ mvn pom.xml test

Using heron-starter with IntelliJ IDEA

Importing heron-starter as a project in IDEA

The following instructions will import heron-starter as a new project in IntelliJ IDEA.

  • Open File > Import Project... and navigate to the top-level directory of your heron-starter clone (e.g. ~/${YOUR-PATH}/heron-starter).
  • Select Import project from external model, select "Maven", and click Next.
  • In the following screen, enable the checkbox Import Maven projects automatically. Leave all other values at their defaults. Click Next.
  • Click Next on the following screen about selecting Maven projects to import.
  • Select the JDK to be used by IDEA for heron-starter, then click Next.
    • At the time of this writing you should use JDK 6.
    • It is strongly recommended to use Sun/Oracle JDK 7 rather than OpenJDK 7.
  • You may now optionally change the name of the project in IDEA. The default name suggested by IDEA is "heron-starter". Click Finish once you are done.