Error Using HdfsLocalCluster

[AvihayTsayeg]

Hi,
I'm using These dependencies:

   "com.github.sakserv" % "hadoop-mini-clusters" % "0.1.8" 
   "com.github.sakserv" % "hadoop-mini-clusters-common" % "0.1.8" 
   "com.github.sakserv" % "hadoop-mini-clusters-hdfs" % "0.1.8"

running simple example getting the following error:

        HdfsLocalCluster hdfsLocalCluster = new HdfsLocalCluster.Builder()
                .setHdfsNamenodePort(12345)
                .setHdfsNamenodeHttpPort(12341)
                .setHdfsTempDir("embedded_hdfs")
                .setHdfsNumDatanodes(1)
                .setHdfsEnablePermissions(false)
                .setHdfsFormat(true)
                .setHdfsEnableRunningUserAsProxyUser(true)
                .setHdfsConfig(new Configuration())
                .build();

        hdfsLocalCluster.start();

I'm getting the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/MiniDFSCluster$Builder
at com.github.sakserv.minicluster.impl.HdfsLocalCluster.start(HdfsLocalCluster.java:173)

Regards, Avihay

[sakserv]

@AvihayTsayeg - sorry for the delay. I broke notifications for this project and missed this issue.

I just put together a simple test using what you've provided above and all seems to work as expected using maven.

I'm not well versed in gradle, but are the transitive dependencies not pulled in? Do you see the hadoop-hdfs jar in your classpath? You might need to explicitly add them.

Here is what hadoop-mini-clusters-hdfs depends on:

        <!-- Hadoop Client -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <!-- Hadoop MiniCluster -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-minicluster</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

Let me know if you have additional details and thanks for the report.

[AvihayTsayeg]

Hi,
Thanks for the tip,
I had the jar in the dependency, but I had to define it as test jar
classifier "tests"

 "org.apache.hadoop" % "hadoop-hdfs" % hadoopLibVersion % "test" classifier "tests"