Hadoop on AARCH64
This documents my process of deploying Hadoop and friends on aarch64 system.
Environment
- Ubuntu 20.04
- Raspberry Pi 4 4GB
Package Versions
- Hadoop 2.10.0
- Hive 1.2.2
- OpenJDK 8
Process
Hadoop
Building native extensions
- Dependencies
- Protobuf 2.5
- Hadoop
Install Dependencies
sudo apt-get install -y libsnappy-dev net-tools maven cmake unzip curl zip ssh vim-tiny pdsh openjdk-8-jdk zlib1g-dev libssl-dev autoconf automake libtool make g++ libsasl2-dev pkg-config
sudo apt-get install build-essential
Protobuf
- Protobuf 2.5 does not support building on aarch64
- Apply the following patch to workaround
- https://gist.github.com/BennettSmith/7111094
Patch Hadoop 2.10.0
Hadoop 2.x does not build with OpenSSL 1.1.1 (only OpenSSL 1.0.2, which is EOL, not installable on ubuntu 20.04)
Configuration
- Copy the configuration files into place
Hive
- Need MySQL 5.7 as metastore
MySQL 5.7
- Ubuntu 20.04 apt only has MySQL 8.0
- Install via Docker
- https://scito.ch/content/mysql-57-docker-container-raspberry-pi-using-debian-sid
References
- https://qiita.com/thashi/items/f8884b78df3faa5ff887
- https://scito.ch/content/mysql-57-docker-container-raspberry-pi-using-debian-sid
- https://www.linode.com/docs/databases/hadoop/how-to-install-and-set-up-hadoop-cluster/#architecture-of-a-hadoop-cluster
- https://github.com/dvoros/docker-hive
- https://dev.to/awwsmm/building-a-raspberry-pi-hadoop-spark-cluster-8b2