apache/seatunnel

[Bug] [Module Name] HADOOP_HOME and hadoop.home.dir are unset

Opened this issue · 5 comments

huchao-hu commented

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

image
使用 seatunnel-engine-examples模块测试 mysql cdc 同步到clickhouse 会报错说没有 HADOOP_HOME and hadoop.home.dir are unset

SeaTunnel Version

2.3.8版本-release 源码

SeaTunnel Config

seatunnel:
    engine:
        backup-count: 1
        queue-type: blockingqueue
        print-execution-info-interval: 60
        slot-service:
            dynamic-slot: true
        checkpoint:
            interval: 300000
            timeout: 10000
            storage:
                type: hdfs
                max-retained: 3
                plugin-config:
                    namespace: /tmp/seatunnel/checkpoint_snapshot/
                    storage.type: local
                    fs.defaultFS: file:///tmp/
        jar-storage:
            enable: false
            connector-jar-storage-mode: SHARED
            connector-jar-storage-path: ""
            connector-jar-cleanup-task-interval: 3600
            connector-jar-expiry-time: 600
        telemetry:
            metric:
                enabled: false

Running Command

直接在idea mian方法启动的

Error Exception

Caused by: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
	at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:549)
	at org.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:570)
	at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:593)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:690)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:78)
	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1665)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.initStorage(HdfsStorage.java:68)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.<init>(HdfsStorage.java:57)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.common.HdfsFileStorageInstance.getOrCreateStorage(HdfsFileStorageInstance.java:53)
	at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorageFactory.create(HdfsStorageFactory.java:74)
	at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.<init>(CheckpointManager.java:105)
	at org.apache.seatunnel.engine.server.master.JobMaster.initCheckPointManager(JobMaster.java:288)
	at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:271)
	at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$4(CoordinatorService.java:511)
	at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:39)

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

hawk9821 commented

Please install hadoop locally

huchao-hu commented

Isn't there a local mode to boot? Do you still need to install hadoop? @hawk9821

fcb-xiaobo commented

Isn't there a local mode to boot? Do you still need to install hadoop? @hawk9821

This shouldn't affect usage, right? The task can still run normally

huchao-hu commented

@fcb-xiaobo Currently, the startup fails and cannot be used normally

hawk9821 commented

Isn't there a local mode to boot? Do you still need to install hadoop? @hawk9821

The error log is so obvious that the problem with the Hadoop Client environment can be solved by following the log prompts.