Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Can't achieve load distribution


Copy link to this message
-
Can't achieve load distribution
Hi,

I have a simple MR job, and I want each Mapper to get one line from my
input file (which contains further instructions for lengthy processing).
Each line is 100 characters long, and I tell Hadoop to read only 100 bytes,

job.getConfiguration().setInt("mapreduce.input.linerecordreader.line.maxlength",
100);

I see that this part works - it reads only one line at a time, and if I
change this parameter, it listens.

However, on a cluster only one node receives all the map tasks. Only one
map tasks is started. The others never get anything, they just wait. I've
added 100 seconds wait to the mapper - no change!

Any advice?

Thank you. Sincerely,
Mark
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB