Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Question about disk space allocation in hadoop


Copy link to this message
-
Question about disk space allocation in hadoop
Yu Li 2010-06-29, 04:32
Hi all,

As we all know, machines in hadoop cluster may be both datanode and
tasktracker, so one machine may store both MR job intermediate data
and HDFS data. My question is: if we have more than one disk per node,
say 4 disks, and would like both job intermediate data and HDFS data
store into all disks to reduce IO times of each single disk, can we
draw a line between space of local FS and HDFS? For example, restrict
the intermediate temp data occupy no more than 25% space on each disk?
Thanks in advance.

Best Regards,
Carp