/insightedge-examples

InsightEdge Examples

Primary LanguageScala

InsightEdge Examples

InsightEdge GigaSpaces convergence platform

Introduction

InsightEdge is a Hybrid transactional/analytical processing platform built on top of Spark and GigaSpaces Data Grid.

This project contains examples of how standard Spark applications can use Data Grid for different purposes.

Requirements

  • Java 1.8
  • Scala 2.10
  • Maven 3.1+
  • SBT 0.13 (optional)
  • InsightEdge distribution

Building project

InsightEdge jars are not published to Maven Central Repository yet. To install artifacts to your local Maven repository, make sure you have Maven installed and then run:

./sbin/insightedge-maven.sh

This project has both SBT and Maven configurations. You can build it with next commands:

# Maven
mvn clean test package

# SBT
sbt clean test assembly

Running examples

There are several options how you can run examples:

Starting local environment

Prior to executing example application, you have to start the Data Grid and deploy an empty space on it. You can do it using demo mode:

./sbin/insightedge.sh --mode demo

Such command will start next components:

  • Spark master at spark://127.0.0.1:7077 and Spark slave
  • Data Grid manager and two containers with 1G heap each
    • space is deployed with name insightedge-space
    • lookup locator is 127.0.0.1:4174
    • lookup group is insightedge

Running from IDE

You can run examples from your favourite IDE. Every example has a main method, so it can be executed as standard application.

Here is an example of run configuration for SaveRDD for Intellij Idea: IDEA run configuration

With this configuration, example will run on local Spark cluster and save the generated RDD to Data Grid to specified space.

Running from command line

You can build the project and submit examples as Spark applications with the next command:

./bin/insightedge-submit --class {main class name} --master {Spark master URL} \
    {insightedge-examples.jar location} \
    {Spark master URL} {space name} {lookup group} {lookup locator}

For example, SaveRDD can be submitted with the next syntax:

./bin/insightedge-submit --class com.gigaspaces.insightedge.examples.basic.SaveRdd --master spark://127.0.0.1:7077 \
    /home/user/Git/insightedge-examples/target/insightedge-examples.jar \
    spark://127.0.0.1:7077 insightedge-space insightedge 127.0.0.1:4174

Note that running TwitterPopularTags example requires you to pass Twitter app tokens as arguments

Stopping local environment

To stop all InsightEdge components, next command can be executed:

./sbin/insightedge.sh --mode shutdown

Troubleshooting

If you have any troubles running the example applications, please, contact us with: