5GB files are uploaded 0 in size
Opened this issue · 0 comments
Hello,
From the documentation files larger 5GB should be partitioned into separate files. In swift-file-system/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeOutputStream.java I see the following variable:
private long filePartSize = 4768709000L; // files greater than 4.5Gb are divided into parts
So, I am using this way to upload files:
(1)
hadoop distcp2 -D fs.swift.partsize=4000000 /media/hadoop/file5G swift://container1.rc/file5G
(2)
hadoop fs -put /media/hadoop/file5G swift://container1.rc/file5G
Neither of them works. From the log hadoop-hduser-tasktracker-hadoop-master-node.log it looks like the file is being uploaded correctly:
2013-09-25 13:40:34,687 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201309251259_0005_m_000000_0 1.0% 97.3% Copying file:/media/hadoop/file5G to swift://container1.rc/file5G [4.9G/5.0G]
2013-09-25 13:42:27,763 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201309251259_0005_m_000000_0 is in commit-pending, task state:COMMIT_PENDING
2013-09-25 13:42:27,764 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201309251259_0005_m_000000_0 1.0% 97.3% Copying file:/media/hadoop/file5G to swift://container1.rc/file5G [4.9G/5.0G]
2013-09-25 13:42:27,824 INFO org.apache.hadoop.mapred.TaskTracker: Received commit task action for attempt_201309251259_0005_m_000000_0
2013-09-25 13:42:28,788 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201309251259_0005_m_000000_0 1.0% 100.0% Copying file:/media/hadoop/file5G to swift://container1.rc/file5G [5.0G/5.0G]
2013-09-25 13:42:28,790 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201309251259_0005_m_000000_0 is done.
Files which are less than 5GB are uploaded just fine. I have tried number of different parameters, options, etc.
As far as I understood, partitioning should work just from the box. So, maybe I am missing something.
I would appreciate if you shed some light how it indeed should work.