sequenceiq/hadoop-docker

Data volumes for persistence and connect to Hive

darrenhaken opened this issue · 2 comments

darrenhaken commented

I'm new to the Hadoop stack so forgive me if I'm missing something obvious.

I had two requirements I'm trying to work out with this Docker image.

  1. how to persist hdfs to a data volume (is hdfs running?)
  2. how to connect another container running another part of the Hadoop stack i.e Hive.

Can anyone help?

patrickneubauer commented

Dear darrenhaken,

we face similar requirements and wonder if you were able to resolve yours.

If so, could you please point us towards the direction or steps that you chose to configure the Hive container to use the hdfs running within this Docker image?

Cheers, Patrick

Mehran91z commented

Hi, I have same problem for using volume for my hdfs input/output.

I want to make directory by $HADOOP_PREFIX/bin/hadoop fs -mkdir mytest and then put files on mytest/input and do something on them like wordcount and I want to persist input and output data after each docker run!

How is it possible?

I made so far:

  1. Added these codes to hdfs-site.xml:
    <property> <name>dfs.datanode.data.dir</name> <value>file:///home/app/hdfs/datanode</value> <description>DataNode directory</description> </property>

  2. Create a docker volume with name 'myvol'

  3. Use -v for run image:
    docker run -v myvol:/home/app -it c29b621ba74a /etc/bootstrap.sh -bash

But in /home/app directory there are just my created files by vi command and another folder named 'hdfs', that not working to persist input/output data.