quantcast/qfs

Error when running mapreduce on QFS

Closed this issue · 0 comments

fsareshwala commented

GitHub user [bofuuchicago|https://github.com/bofuuchicago] has created an issue on our public [GitHub Issues|https://github.com/quantcast/qfs/issues] list.

This ticket is a mirror of the first post of the public ticket on github. The purpose of this ticket is to have an owner of the public response. The owner is responsible for researching the answer and responding to the public issue in a timely manner.

#60

Issue Description
{code}
Hi,

I have an issue when running mapreduce on qfs.
Everything works just fine when using bin/hadoop fs commands.
When I tried to run wordcount, it says:

15/06/24 09:48:40 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:18040
15/06/24 09:48:41 INFO input.FileInputFormat: Total input paths to process : 1
15/06/24 09:48:41 INFO mapreduce.JobSubmitter: number of splits:1
15/06/24 09:48:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1435157289542_0001
15/06/24 09:48:41 INFO impl.YarnClientImpl: Submitted application application_1435157289542_0001
15/06/24 09:48:41 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1435157289542_0001/
15/06/24 09:48:41 INFO mapreduce.Job: Running job: job_1435157289542_0001
15/06/24 09:48:47 INFO mapreduce.Job: Job job_1435157289542_0001 running in uber mode : false
15/06/24 09:48:47 INFO mapreduce.Job: map 0% reduce 0%
15/06/24 09:48:52 INFO mapreduce.Job: Task Id : attempt_1435157289542_0001_m_000000_0, Status : FAILED
Exception from container-launch: ExitCodeException exitCode=1:
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

15/06/24 09:48:57 INFO mapreduce.Job: Task Id : attempt_1435157289542_0001_m_000000_1, Status : FAILED
Exception from container-launch: ExitCodeException exitCode=1:
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

15/06/24 09:49:02 INFO mapreduce.Job: Task Id : attempt_1435157289542_0001_m_000000_2, Status : FAILED
Exception from container-launch: ExitCodeException exitCode=1:
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

15/06/24 09:49:09 INFO mapreduce.Job: map 100% reduce 100%
15/06/24 09:49:09 INFO mapreduce.Job: Job job_1435157289542_0001 failed with state FAILED due to: Task failed task_1435157289542_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

15/06/24 09:49:09 INFO mapreduce.Job: Counters: 12
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=11945
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=11945
Total vcore-seconds taken by all map tasks=11945
Total megabyte-seconds taken by all map tasks=12231680
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0

The logs for container is:
/staging/fubo/.staging/job_1435157289542_0002/job_1435157289542_0002_1.jhist to qfs://localhost:20000/tmp/hadoop-yarn/staging/history/done_intermediate/fubo/job_1435157289542_0002-1435157856115-fubo-word+mean-1435157882595-0-0-FAILED-default-1435157860419.jhist_tmp
2015-06-24 09:58:02,623 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: qfs://localhost:20000/tmp/hadoop-yarn/staging/history/done_intermediate/fubo/job_1435157289542_0002-1435157856115-fubo-word+mean-1435157882595-0-0-FAILED-default-1435157860419.jhist_tmp
2015-06-24 09:58:02,623 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying qfs://localhost:20000/tmp/hadoop-yarn/staging/fubo/.staging/job_1435157289542_0002/job_1435157289542_0002_1_conf.xml to qfs://localhost:20000/tmp/hadoop-yarn/staging/history/done_intermediate/fubo/job_1435157289542_0002_conf.xml_tmp
2015-06-24 09:58:02,631 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: qfs://localhost:20000/tmp/hadoop-yarn/staging/history/done_intermediate/fubo/job_1435157289542_0002_conf.xml_tmp
2015-06-24 09:58:02,632 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: qfs://localhost:20000/tmp/hadoop-yarn/staging/history/done_intermediate/fubo/job_1435157289542_0002.summary_tmp to qfs://localhost:20000/tmp/hadoop-yarn/staging/history/done_intermediate/fubo/job_1435157289542_0002.summary
2015-06-24 09:58:02,632 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: qfs://localhost:20000/tmp/hadoop-yarn/staging/history/done_intermediate/fubo/job_1435157289542_0002_conf.xml_tmp to qfs://localhost:20000/tmp/hadoop-yarn/staging/history/done_intermediate/fubo/job_1435157289542_0002_conf.xml
2015-06-24 09:58:02,633 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: qfs://localhost:20000/tmp/hadoop-yarn/staging/history/done_intermediate/fubo/job_1435157289542_0002-1435157856115-fubo-word+mean-1435157882595-0-0-FAILED-default-1435157860419.jhist_tmp to qfs://localhost:20000/tmp/hadoop-yarn/staging/history/done_intermediate/fubo/job_1435157289542_0002-1435157856115-fubo-word+mean-1435157882595-0-0-FAILED-default-1435157860419.jhist
2015-06-24 09:58:02,633 INFO [Thread-53] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2015-06-24 09:58:02,636 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to Task failed task_1435157289542_0002_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

2015-06-24 09:58:02,638 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://localhost:19888/jobhistory/job/job_1435157289542_0002
2015-06-24 09:58:02,645 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
2015-06-24 09:58:03,646 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:1 RackLocal:0
2015-06-24 09:58:03,648 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory qfs://localhost:20000 /tmp/hadoop-yarn/staging/fubo/.staging/job_1435157289542_0002
2015-06-24 09:58:03,650 INFO [Thread-53] org.apache.hadoop.ipc.Server: Stopping server on 55169
2015-06-24 09:58:03,653 INFO [IPC Server listener on 55169] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 55169
2015-06-24 09:58:03,653 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2015-06-24 09:58:03,653 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted

Can anyone tell me what's the problem? Please tell me if I need to provide more information about it
{code}