Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Hadoop Mapper Intermediate Result Storage with No Reducer


Copy link to this message
-
Hadoop Mapper Intermediate Result Storage with No Reducer
Hi,

We are running Hadoop jobs with  mappers only, no reducers.
After each map computation, it writes its result (i.e. intermediate
results) to local disk. How fast is this speed, compared with write-speed
to HDFS?
We used TestFDSIO to do the benchmark to get write-speed to HDFS, but we
wonder how much the gap is.

Another question is, when did Hadoop move this chuck to HDFS?

Any thoughts, guys? Thanks ahead.

Xun
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB