HDP Test Examples

The goal of this project is to create unit test examples for common HDP libraries. We will be focusing on Pig, Spark, HBase, Hive, and Hadoop. It is the goal of this project to supply tests in Scala and Java.

Running the Tests

Below is an example of how to run the current test suite using a given profile. The profile hdp-2.4.2 is used in the example below.

$ mvn clean test -P hdp-2.4.2

Testing Libraries that Are Used

JUnit
Mockito
ScalaTest
Spark Testing Base. Its source can be found here.
PigUnit. This link points to version 0.15.0 of pigunit.

Test Data

Using sensor data from this tutorial

Setting up Kerberos Enabled Cluster

Creating cluster (Summarizing this tutorial)

Clone Ambari Vagrant Repository and Create the 3 Nodes

git clone https://github.com/u39kun/ambari-vagrant.git
cd ambari-vagrant/centos6.4
./up.sh 3

Once this is Complete

vagrant ssh c6401
sudo su -
wget -O /etc/yum.repos.d/ambari.repo http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.1.0/ambari.repo
yum install ambari-server -y
ambari-server setup -s
ambari-server start

Setting up the Cluster via Ambari:
- Go to: c6401.ambari.apache.org:8080
- Login using:
  - username: admin
  - password: admin
- Click 'Launch Install Wizard'
- Type in a name that you want to call this cluster. Hit next.
- Choose HDP version. This is based on HDP 2.4. Click Next
- In host names add: c64[01-03].ambari.apache.org
- Select the file named 'insecure_private_key'
- SSH User account should be 'root'
- Click 'Register and Confirm'. The 3 nodes you created should now be listed.
- Click OK
- Click Next once host confirmation is successful.
- Continue through the wizard. Install components you'd like to use.
- Once cluster is setup continue to the next step to enable kerberos.

Setting Up Kerberos

Install necessary Kerberos tools and create database.

yum install krb5-server krb5-libs krb5-auth-dialog rng-tools -y
rngd -r /dev/urandom -o /dev/random
/usr/sbin/kdb5_util create -s

Update '/etc/krb5.conf'. Example below.

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = HORTONWORKS.LOCAL
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5


[domain_realm]

  hortonworks.local = HORTONWORKS.LOCAL

  .hortonworks.local = HORTONWORKS.LOCAL



[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log

[realms]
  HORTONWORKS.LOCAL = {
    admin_server = c6401.ambari.apache.org
    kdc = c6401.ambari.apache.org
  }

Restart Services

/etc/rc.d/init.d/krb5kdc restart
/etc/rc.d/init.d/kadmin restart

Adding admin principal

sudo kadmin.local
kadmin.local:  add_principal admin/admin@EXAMPLE.COM
WARNING: no policy specified for admin/admin@EXAMPLE.COM; defaulting to no policy
Enter password for principal "admin/admin@EXAMPLE.COM":
Re-enter password for principal "admin/admin@EXAMPLE.COM":
Principal "admin/admin@EXAMPLE.COM" created.

Enabling Kerberos in Ambari

Creating Keytab

kadmin
addprinc jj@EXAMPLE.COM
ktutil
addent -password -p jj -k 1 -e RC4-HMAC
wkt jj.keytab
q

Retrieving Necessary Files

Replace the following files with files obtained from the cluster just created:
- src/main/resources/core-site.xml
- src/main/resources/hbase-site.xml
- src/main/resources/hdfs-site.xml
- src/main/resources/jj.keytab
- src/main/resources/krb5.conf

jjmeyer0/hdp-test-examples