/hadoop

Primary LanguageShell

Hadoop Example

  • ResourceManager and NodeManager will be running with docker on k8s.
  • HDFS is used for test, now it is running in docker with single node mode.

Cluster init

Checkout: (init/README.md)

Usage

step0 Prerequisites

  • Installed kubenetes cluster (or single node).
  • kubectl command line tool.

step1 Deploy HDFS

	kubectl create -f hadoop/hadoop-hdfs-rc.yaml
	kubectl create -f hadoop/hadoop-hdfs-svc.yaml

step2 Deploy Zookeeper

	kubectl create -f hadoop/zookeeper-rc.yaml
	kubectl create -f hadoop/zookeeper-svc.yaml

step2 Deploy Hadoop ResourceManager

  • Get HDFS service IP
yancey@ yancey-macbook hadoop$kubectl get svc
NAME            CLUSTER-IP   EXTERNAL-IP   PORT(S)                                                 AGE
hadoop-hdfs     10.0.0.114   <none>        9000/TCP,50070/TCP                                      50m
hadoop-master   10.0.0.246   <none>        8088/TCP,8033/TCP,8032/TCP,8031/TCP,8030/TCP,8090/TCP   3d
kubernetes      10.0.0.1     <none>        443/TCP                                                 7d
zookeeper       10.0.0.70    <none>        2181/TCP                                                2d
  • Generate hadoop-master-rc.yaml from hadoop-master-rc.yaml.default
sed "s/<ZOOKEEPER_ADDR>/10.0.0.70:2181/g" hadoop-master-rc.yaml.default | sed "s/<HDFS_HOST>/10.0.0.114/g" > hadoop-master-rc.yaml
  • Create hadoop-master pod and service.
kubectl create -f hadoop/hadoop-master-rc.yaml
kubectl create -f hadoop/hadoop-master-svc.yaml

step3 Deploy Hadoop NodeManager

  • Generate hadoop-slave-rc.yaml from hadoop-slave-rc.yaml.default
sed "s/<RM_HOSTNAME>/10.0.0.246/g" hadoop-slave-rc.yaml.default | sed "s/<HDFS_HOST>/10.0.0.114/g" > hadoop-slave-rc.yaml
  • Create hadoop node manager pod
kubectl create -f hadoop/hadoop-slave-rc.yaml
  • append hadoop-slave hostname to the /etc/hosts on hadoop master pod.
yancey@ yancey-macbook hadoop$kubectl get pods
NAME                        READY     STATUS    RESTARTS   AGE
hadoop-hdfs-hay0j           1/1       Running   0          19h
hadoop-master-ir8sl         1/1       Running   0          4h
hadoop-slave-yxgt5          1/1       Running   0          4h

kubectl exec -it hadoop-master-ir8sl /bin/bash
root@hadoop-master-ir8sl:/opt/hadoops#echo "10.1.99.2       hadoop-slave-yxgt5" >> /etc/hosts

step4 Run a MR job

  • Exec to hadoop-master pod
root@hlg-2p29-yanxu:~# kubectl exec -it hadoop-master-ir8sl /bin/bash
  • Run a mapreduce job
root@hadoop-master-ir8sl:/opt/hadoops# pwd
/opt/hadoops
root@hadoop-master-ir8sl:/opt/hadoops# sh run_wordcount.sh

  • Resource Manager web

    You can open resource manager web with the address of the node address which hadoop-master pod running. like:

    http://{hadoop-master-node}:18088/cluster/apps