Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> dfs.datanode.du.reserved


Copy link to this message
-
Re: dfs.datanode.du.reserved
Hey John,

Ideas, comments and patches are welcome on
https://issues.apache.org/jira/browse/HDFS-1564 for achieving this!
On Wed, Mar 6, 2013 at 9:56 PM, John Meza <[EMAIL PROTECTED]> wrote:

> Thanks for the reply. This sounds like it has potential, but also seems to
> be a rather duct-tape type of work around. It would be nice if there was a
> mod to dfs.datanode.du.reserved that worked within Hadoop, so that would
> imply that hadoop was a little more certain to adhere to it.
>
> I understand that dfs.datanode.du.reserved defines reservd storage on each
> volume. I would like to give each volume a different reserved value.
>
> An example:
>
>  <name>dfs.datanode.du.reserved</name>
>    <value>///hstore1/dfs/dn:161061273600,///hstore2/dfs/dn:53687091200</value>
>
> Or something similiar.
> thanks
> John
> ------------------------------
> Date: Wed, 6 Mar 2013 10:25:17 +0100
> Subject: Re: dfs.datanode.du.reserved
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
>
> Not that I know. If so you should be able to identify each volume and as
> of now this isn't the case.
> BUT it can be done without Hadoop knowing about it, at the OS level, using
> different partitions/mounts for datanode and jobtracker stuff. That should
> solve your problem.
>
> Regards
>
> Bertrand
>
> On Mon, Mar 4, 2013 at 10:26 PM, John Meza <[EMAIL PROTECTED]> wrote:
>
> I'm probably not being clear.
> this seems to describe it: dfs.datanode.du.reserved  configured
> per-volume.
> https://issues.apache.org/jira/browse/HDFS-1564
>
> thanks
> John
> ------------------------------
> From: [EMAIL PROTECTED]
> Date: Mon, 4 Mar 2013 15:37:36 -0500
> Subject: Re: dfs.datanode.du.reserved
> To: [EMAIL PROTECTED]
>
>
> Possible to reserve 0 from various testing I have done yet that could
> cause the obvious side effect of achieving zero disk space:) Have only
> tested in development environment, however. Yet there are various tuning
> white papers and other benchmarks where the very same has been tested.
>
> Thanks.
>
> On Mon, Mar 4, 2013 at 2:00 PM, John Meza <[EMAIL PROTECTED]> wrote:
>
> the parameter: dfs.datanode.du.reserved is used to reserve disk space PER
> datanode. Is it possible to reserve a different amount of disk space per
> DISK?
>
> thanks
> John
>
>
>
>
> --
> Ellis R. Miller
> 937.829.2380
>
> <http://my.wisestamp.com/link?u=2hxhdfd4p76bkhcm&site=www.wisestamp.com/email-install>
>
> Mundo Nulla Fides
>
>
>
> <http://my.wisestamp.com/link?u=gfbmwhzrwxzcrjqx&site=www.wisestamp.com/email-install>
>
>
>
>
>
>
--
Harsh J