/\
/ \ /\
/ /\ / \
_ _ / / / /\ \
/_\ _ __ __ _ | |_ ___ _ _ __(_( _(_(_ )_)
/ _ \ | ' \ / _` || _|/ -_)| '_|/ _` |(_-<| || |
/_/ \_\|_|_|_|\__,_| \__|\___||_| \__,_|/__/ \_,_|
Apache Amaterasu is an open-source, deployment tool for data pipelines. Amaterasu allows developers to write and easily deploy data pipelines, and clusters manage their configuration and dependencies.
For this preview version, we have packaged amaterasu nicely for you to just download and extract. Once you do that, you are just a couple of easy steps away from running your first job.
We have also created a Mesos cluster you can use to test Amaterasu or use for development purposes. For more details, visit the amaterasu-vagrant repo
Configuring amaterasu is very simple. Before running amaterasu, open the amaterasu.properties file in the top-level amaterasu directory, and verify the following properties:
| property | Description | Default value |
|---|---|---|
| zk | The ZooKeeper connection string to be used by amaterasu |
192.168.33.11 |
| master | The clusters' Mesos master | 192.168.33.11 |
| user | The user that will be used to run amaterasu |
root |
To run an amaterasu job, run the following command in the top-level amaterasu directory:
ama-start.sh --repo="https://github.com/shintoio/amaterasu-job-sample.git" --branch="master" --env="test" --report="code"
We recommend you either fork or clone the job sample repo and use that as a starting point for creating your first job.
As part of the ASFs policy, when building Apache Amaterasu from a source distribution and not from git, the gradle-wrapper.jar needs to be downloaded. The following command, will download the gradle wrapper from Amaterasu git repository and puts under incubator-amaterasu/gradle/wrapper
wget --no-check-certificate -P incubator-amaterasu/gradle/wrapper https://github.com/apache/incubator-amaterasu/raw/version-0.2.0-incubating-rc4/gradle/wrapper/gradle-wrapper.jar (or) curl --insecure -L https://github.com/apache/incubator-amaterasu/raw/version-0.2.0-incubating-rc4/gradle/wrapper/gradle-wrapper.jar > incubator-amaterasu/gradle/wrapper/gradle-wrapper.jar
to build the amaterasu home dir (for dev purposes) run:
./gradlew buildHomeDir test
to create a distributable jar (clean creates the home dir first) run:
./gradlew buildDistribution test
Amaterasu is an Apache Mesos framework with two levels of schedulers:
- The ClusterScheduler manages the execution of all the jobs
- The JobScheduler manages the flow of a job
The main clases in Amateraso are listed bellow:
+-------------------------+ +------------------------+
| ClusterScheduler | | Kami |
| |-->| |
| Manage jobs: | | Manages the jobs queue |
| Queue new jobs | | and Amaterasu cluster |
| Reload interrupted jobs | +------------------------+
| Monitor cluster state |
+-------------------------+
|
| +------------------------+
| | JobExecutor |
| | |
+---->| Runs the Job Scheduler |
| Communicates with the |
| ClusterScheduler |
+------------------------+
|
|
+------------------------+ +---------------------------+
| JobScheduler | | JobParser |
| | | |
| Manages the execution |----->| Parses the kami.yaml file |
| of the job, by getting | | and create a JobManager |
| the execution flow | +---------------------------+
| fron the JobManager | |
| and comunicating with | +---------------------------+
| Mesos | | JobManager |
+------------------------+ | |
| | Manages the jobs workflow |
| | independently of mesos |
+------------------------+ +---------------------------+
| ActionExecutor |
| |
| Executes ActionRunners |
| and manages state for |
| the executor |
+------------------------+