Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> io.sort.factor


Hi,

io.sort.factor  --  The number of streams to merge at once while sorting files. This determines the number of open file handles.
How can I use this parameter to improve performance of mapreduce job?
My understanding from above description was If there are many spill records then increasing io.sort.mb as well as io.sort.factor will help in better performance. Increasing io.sort.mb helped but changing io.sort.factor (> 10) does not seem to improve/degrade performance of mapred  job.

Regards,
Ajay Srivastava
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB