/hadoop-2.7.2-src

hadoop version 2.7.2

Primary LanguageJavaApache License 2.0Apache-2.0

# hadoop-2.7.2-src
hadoop version 2.7.2
环境: MacOS + IntelliJ IDEA
软件包:
protobuf-2.5.0.tar.gz

macos环境变量配置:

在.bash_profile 添加如下环境变量:

#关闭Homebrew 自动更新
export HOMEBREW_NO_AUTO_UPDATE=true

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/

export ANT_HOME=/opt/majorlu/github/Hadoop_code/apache-ant-1.9.14
export M2_HOME=/opt/majorlu/github/Hadoop_code/apache-maven-3.6.3

export OPENSSL_ROOT_DIR=/usr/local/opt/openssl
export OPENSSL_INCLUDE_DIR=$OPENSSL_ROOT_DIR/include
export PATH=$PATH:$M2_HOME/bin:$ANT_HOME/bin

CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

PATH=$JAVA_HOME/bin:$PATH:

export JAVA_HOME

export CLASSPATH

1、进入Hadoop 2.7.2源代码命令,执行如下命令编译:
mvn clean package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true

2、hadoop-hdfs的把webapps拷贝到hadoop-hdfs/src/main/resources/下,
拷贝resourcemanager项目的src/test/resources/log4.properties拷贝到src/main/resources/

3、IDEA下设置: File-->Project Structure--->Modules设置:
[把以下的依赖项改成Compile级别]
hadoop-common
commons-configuration:commons-configuration:1.6
hadoop-auth(有两个都要改)
org.apache.curator:curator-framework:2.6.0
org.apache.curator:curator-client:2.6.0
#启动resourcemanager后发现访问http://localhost:8088报错
继续把org.apache.httpcomponents:httpclient:4.2.5改成Compile

4、配置文件设置:

hdfs-default.xml:

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://localhost:9000</value>
</property>

core-site.xml:

core-site.xml:
<property>
  <name>hadoop.tmp.dir</name>
  <value>/opt/majorlu/github/Hadoop_code/tmp</value>
  <description>A base for other temporary directories.</description>
</property>

5、启动:
运行namenode时使用-format先格式化hdfs文件系统
再次启动namenode不要使用-format参数
启动datanode进程
启动ResourceManager进程