yahoo/storm-yarn

Proposal: Merge storm-yarn into storm source tree

clockfly opened this issue · 3 comments

clockfly commented

I think it is a good idea to merge storm-yarn into storm source tree. This approach have been adopted by spark. spark-yarn is a internal part of spark.

Advantages:

  1. More easy to manage the dependance.
  2. We can finally throw away the zmq code and dependance for build and test.
  3. In storm-yarn, we don't need to install both, don't need to manage STORM_HOME in storm-yarn anymore. There will be single installation, and this can be build into single rpm package. Very convenient for management.
  4. No longer need local storm.zip package. We can set the jars(storm.jar storm-yarn.jar) and libraries as local resources, and use this to start container on another machine.

Steps:

  1. First need to change the groupId to "storm"
  2. Better we need to use pom to manage storm, so that we can more easily aggregate storm-yarn as a module.
  3. Add interface in storm-yarn scripts, so that we can submit storm job with storm-yarn script file directly (no need to getStormConfig)
  4. We should not try to copy storm.zip into HDFS and use that zip to deploy. Instead, we can just add lib and jars into local resource. Otherwise, there are two copy of storm. One is in STORM_HOME, the other is in storm.zip.
clockfly commented

After merging, the binary layout of storm is like this:

/etc/storm/conf
/usr/bin/storm-yarn --> /usr/lib/storm/bin/storm-yarn
/usr/lib/storm/
Storm-core.jar
Storm-netty.jar
Storm-yarn.jar
/usr/lib/storm/bin/
Storm-yarn
Storm
/usr/lib/storm/lib/
dependant-xx.jar
..
dependant-yy.jar

I think the layout is much cleaner than current storm-yarn deployment.

revans2 commented

Totally agree. Storm has a proposal up to become an Apache incubator project. The plan was to wait until it is part of the incubator and then work with Nathan about brining in it.

anfeng commented

I don't think that storm-yarn should be merged into storm at this point. Our plan is to have storm-yarn and storm-mesos as a subproject in apache

Andy Feng

Sent from my iPhone

On Sep 18, 2013, at 9:36 AM, Sean Zhong notifications@github.com wrote:

After merging, the binary layout of storm is like this:

/etc/storm/conf
/usr/bin/storm-yarn --> /usr/lib/storm/bin/storm-yarn
/usr/lib/storm/
Storm-core.jar
Storm-netty.jar
Storm-yarn.jar
/usr/lib/storm/bin/
Storm-yarn
Storm
/usr/lib/storm/lib/
dependant-xx.jar
..
dependant-yy.jar

I think the layout is much cleaner than current storm-yarn deployment.

—
Reply to this email directly or view it on GitHub.