/certain-hadoop-formula

Salt formula for Hadoop

Primary LanguageSaltStack

Hadoop

A Salt formula to install hadoop

Currently supports HDFS and YARN

Dependencies:

Minimal Configuration:

hadoop:
  hdfs:
    name_node:
      target: 'hdfs01*'
    data_node:
      target: 'hdfs*'
  yarn:
    resource_manager:
      target: 'yarn01*'
    node_manager:
      target: 'yarn*'

Bootstrap:

Passing the bootstrap pillar tells the NameNode and ResourceManager to store their endpoints in SDB and tells the NameNode to format HDFS.

WARNING: Passing this option will reformat HDFS which is a destructive operation. Only do this if you are comfortable losing data.

salt '<tgt>' state.apply hadoop pillar='{"bootstrap": "True"}'

Available States:

  • hadoop
    • Install Hadoop YARN and HDFS
  • hadoop.install
    • Install user and packages
  • hadoop.hdfs
    • Install Hadoop HDFS
  • hadoop.hdfs.configure
    • Manage hdfs-site.xml
  • hadoop.hdfs.name_node
    • Install Hadoop HDFS NameNode
  • hadoop.hdfs.name_node.install
    • Manage service and environment file, and format HDFS
  • hadoop.hdfs.name_node.service
    • Start the NameNode service
  • hadoop.hdfs.name_node.store_name_node_endpoint
    • Store the NameNode endpoint in SDB
  • hadoop.hdfs.data_node
    • Install Hadoop HDFS DataNode
  • hadoop.hdfs.data_node.install
    • Manage service and environment file
  • hadoop.hdfs.data_node.service
    • Start the NameNode service
  • hadoop.yarn
    • Install Hadoop YARN
  • hadoop.yarn.configure
    • Manage yarn-site.xml
  • hadoop.yarn.resource_manager
    • Install Hadoop YARN ResourceManager
  • hadoop.yarn.resource_manager.install
    • Manage service and environment file
  • hadoop.yarn.resource_manager.service
    • Start the ResourceManager service
  • hadoop.yarn.resource_manager.store_resource_manager_endpoint
    • Store the ResourceManager endpoint in SDB
  • hadoop.yarn.node_manager
    • Install Hadoop YARN NodeManager
  • hadoop.yarn.node_manager.install
    • Manage service and environment file
  • hadoop.yarn.node_manager.service
    • Start the NodeManager service