Hi.

 

We're running a small 30 node cluster  and in a few days will reinstall the
whole software, thus I want to change HDD configuration that was done long
time ago and seems to be inefficient - each node has 2x1TB drives that are
LVMed to one single volume.

 

How do people usually setup drives? For example will it be better to mount
them to two separate folders and feed these folder to both tasktracker and
datanode? Or setup LVM with raid 0 to maximize bandwidth.  

 

What I want is that 2TB of drive space per node were equally accessible to
both tasktracker and datanode, and I'm not sure that mounting two drives to
separate folders achieves that.  (for example if reducer fills one drive
will it start writing the rest of the data to second drive? )

 

Thanks.

For both the DN and TT you can provide a comma separated list of directories.

So, drive 1 could be /hadoop1
And drive 2 /hadoop2

Then in each of those there could be a dfs directory and another for task temp storage.

Hadoop will round-robin writes to these automatically.

Dfs.data.dir might look something like:
<property>
  <name>dfs.data.dir</name>
  <value>/hadoop1/dfs/data,/hadoop2/dfs/data</value>
  <description>Determines where on the local filesystem an DFS data node
  should store its blocks.  If this is a comma-delimited
  list of directories, then data will be stored in all named
  directories, typically on different devices.
  Directories that do not exist are ignored.
  </description>
</property>

And the local mapreduce dir might look like:
<property>
  <name>mapred.local.dir</name>
  <value>/hadoop1/tmp,/hadoop2/tmp</value>
  <description>The local directory where MapReduce stores intermediate
  data files.  May be a comma-separated list of
  directories on different devices in order to spread disk i/o.
  Directories that do not exist are ignored.
  </description>
</property>
On 7/13/09 11:50 AM, "Dmitry Pushkarev" <[EMAIL PROTECTED]> wrote:

Hi.

We're running a small 30 node cluster  and in a few days will reinstall the
whole software, thus I want to change HDD configuration that was done long
time ago and seems to be inefficient - each node has 2x1TB drives that are
LVMed to one single volume.

How do people usually setup drives? For example will it be better to mount
them to two separate folders and feed these folder to both tasktracker and
datanode? Or setup LVM with raid 0 to maximize bandwidth.

What I want is that 2TB of drive space per node were equally accessible to
both tasktracker and datanode, and I'm not sure that mounting two drives to
separate folders achieves that.  (for example if reducer fills one drive
will it start writing the rest of the data to second drive? )

Thanks.

Thanks!  I suspected that there is a RR scheme, but didn't find it anywhere
in documentation.

-----Original Message-----
From: Scott Carey [mailto:[EMAIL PROTECTED]]
Sent: Monday, July 13, 2009 12:39 PM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re: Disk configuration.

For both the DN and TT you can provide a comma separated list of
directories.

So, drive 1 could be /hadoop1
And drive 2 /hadoop2

Then in each of those there could be a dfs directory and another for task
temp storage.

Hadoop will round-robin writes to these automatically.

Dfs.data.dir might look something like:
<property>
  <name>dfs.data.dir</name>
  <value>/hadoop1/dfs/data,/hadoop2/dfs/data</value>
  <description>Determines where on the local filesystem an DFS data node
  should store its blocks.  If this is a comma-delimited
  list of directories, then data will be stored in all named
  directories, typically on different devices.
  Directories that do not exist are ignored.
  </description>
</property>

And the local mapreduce dir might look like:
<property>
  <name>mapred.local.dir</name>
  <value>/hadoop1/tmp,/hadoop2/tmp</value>
  <description>The local directory where MapReduce stores intermediate
  data files.  May be a comma-separated list of
  directories on different devices in order to spread disk i/o.
  Directories that do not exist are ignored.
  </description>
</property>
On 7/13/09 11:50 AM, "Dmitry Pushkarev" <[EMAIL PROTECTED]> wrote:

Hi.

We're running a small 30 node cluster  and in a few days will reinstall the
whole software, thus I want to change HDD configuration that was done long
time ago and seems to be inefficient - each node has 2x1TB drives that are
LVMed to one single volume.

How do people usually setup drives? For example will it be better to mount
them to two separate folders and feed these folder to both tasktracker and
datanode? Or setup LVM with raid 0 to maximize bandwidth.

What I want is that 2TB of drive space per node were equally accessible to
both tasktracker and datanode, and I'm not sure that mounting two drives to
separate folders achieves that.  (for example if reducer fills one drive
will it start writing the rest of the data to second drive? )

Thanks.

Be very cautious with spaces (i.e. don't use them)

On Mon, Jul 13, 2009 at 12:38 PM, Scott Carey <[EMAIL PROTECTED]>wrote:

> For both the DN and TT you can provide a comma separated list of
> directories.
>
> So, drive 1 could be /hadoop1
> And drive 2 /hadoop2
>
> Then in each of those there could be a dfs directory and another for task
> temp storage.
>
> Hadoop will round-robin writes to these automatically.
>
> Dfs.data.dir might look something like:
> <property>
>  <name>dfs.data.dir</name>
>  <value>/hadoop1/dfs/data,/hadoop2/dfs/data</value>
>

--
Ted Dunning, CTO
DeepDyve

111 West Evelyn Ave. Ste. 202
Sunnyvale, CA 94086
http://www.deepdyve.com
858-414-0013 (m)
408-773-0220 (fax)

You may want to add what you learn to
http://wiki.apache.org/hadoop/DiskSetup.

On Mon, Jul 13, 2009 at 12:55 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Be very cautious with spaces (i.e. don't use them)
>
> On Mon, Jul 13, 2009 at 12:38 PM, Scott Carey <[EMAIL PROTECTED]
> >wrote:
>
> > For both the DN and TT you can provide a comma separated list of
> > directories.
> >
> > So, drive 1 could be /hadoop1
> > And drive 2 /hadoop2
> >
> > Then in each of those there could be a dfs directory and another for task
> > temp storage.
> >
> > Hadoop will round-robin writes to these automatically.
> >
> > Dfs.data.dir might look something like:
> > <property>
> >  <name>dfs.data.dir</name>
> >  <value>/hadoop1/dfs/data,/hadoop2/dfs/data</value>
> >
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
> 111 West Evelyn Ave. Ste. 202
> Sunnyvale, CA 94086
> http://www.deepdyve.com
> 858-414-0013 (m)
> 408-773-0220 (fax)
>

On 7/13/09 11:50 AM, "Dmitry Pushkarev" <[EMAIL PROTECTED]> wrote:
> What I want is that 2TB of drive space per node were equally accessible to
> both tasktracker and datanode, and I'm not sure that mounting two drives to
> separate folders achieves that.  (for example if reducer fills one drive
> will it start writing the rest of the data to second drive? )

In practice, we're starting to think that sharing the same file system space
doesn't really work.

What we sometimes see during large disk spills is that the tasks take
significantly more disk space than what was anticipated.  Since the data
node now has less space to work with (if any), the name node won't schedule
writes to those nodes.  This means the name node now searches for free space
and eventually goes into a death spiral, hunting for blocks that do not
violate the locality rules.

We are thinking more and more that putting MR and HDFS on separate file
systems is the way to go when dealing with apps that are unpredictable.
This also helps significantly with capacity planning, as the ops team knows
-exactly- how much space is for HDFS and how much is for MR.  If you use a
file system like ZFS, you can even adjust on the fly if your estimates are
off.