Performance improvements issue using CombineTextInputFormat with many small files
Closed this issue · 12 comments
Hi dgadiraju,
I have tried your code for testing performance improvements using small files on local machine.
Scenario: I have a 100 small files ~30MB each.
Case 1: When map-reduce job was executed with these 100 files rowcount.java, 100 mappers completed the job in ~10 minutes.
Case 2: When the job was executed using RowCountCombinedFileInputFormat.java, 25 mappers were created and the job took ~33 minutes to complete.
I haven't noticed the performance improvements using CombineTextInputFormat.
What may be the possible issues that causing this?
What is the size of the cluster?
This is local env. Pseudo-distributed mode.
8GB Memory. 4 CPU's
It is very tough to see the expected behavior while running in local or pseudo distributed mode. You need to try it out in true cluster. By reducing number of mappers to 30 instead of 100 need not result in reduction in execution time. But it will use fewer resources on large cluster which means more jobs can run simultaneously.
I was just wondering why don't I see performance improvement referring this post that uses CombineFileInputFormat.
Since all other performance metrics discussed on series have shown noticeable performance improvement even on the local machine.
Can you paste counters information of both runs?
Before one is map only job, there are no reducers involved in it. These runs cannot be comparable.
These below lines are in after, but not in before. So they are not functionally same to compare.
Reduce input groups=1
Reduce shuffle bytes=3271558374
Reduce input records=272629852
Reduce output records=1
I haven't made any changes in code. But I see following logs in both of the runs:
before: 16/03/06 21:28:24 INFO mapred.LocalJobRunner: reduce task executor complete
after: 16/03/06 22:10:03 INFO mapred.LocalJobRunner: reduce task executor complete.
Also, job RowCount.java (before) has NoKeyRecordCountReducer reducer. These job looks no different than using combineTextInputFormat in later case.
Thanks for the help!
I am not sure. As mentioned earlier, running local mode gives inconsistent results. At least you need to consider VM and set up full cluster on single node. Best is to use Cloudera or Hortonworks.
The issue is with the execution time since second run took 3 times as compared to first run. Also there were plenty of resources available on local machine while running the job.
There are also chances that the time difference ration would be same as the before job(100 mapper) possibly take less than 10 minutes. If executed on cluster.
I am not able to figure out with the information provided. Counters is not showing the true picture of what you are saying. Clearly "before" is not using reducers as per the counter information.
These does not mean that there is reduce task. If you go through the progress, "before" reduce progress have reached 100% directly where as "after" it became 100% incrementally.
before: 16/03/06 21:28:24 INFO mapred.LocalJobRunner: reduce task executor complete
after: 16/03/06 22:10:03 INFO mapred.LocalJobRunner: reduce task executor complete.
I am not 100% sure about pseudo distributed mode and I always prefer VMs with full cluster on it. Hence I cannot give proper explanation.
I am 100% sure that your comparison is not right. "before" runs as map only and "after" runs using reducers. You have to troubleshoot from that angle and see why "before" is not using reducers while "after" is using reducers.
Hi dgadiraju,
I tried rerunning both of the jobs, the before and after. Now I see the progress of reduce phase incrementally. I have attached the output of runs. So this becomes a valid comparision.
However, the difference in execution time still present(41 minutes & 31 minutes).
Any thoughts on this?
Thanks!
small_files_before_git.txt
small_files_after_git.txt