Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> dfs.datanode.du.reserved


+
John Meza 2013-03-04, 19:00
+
Ellis Miller 2013-03-04, 20:37
+
John Meza 2013-03-04, 21:26
Copy link to this message
-
Re: dfs.datanode.du.reserved
Hey John,

Ideas, comments and patches are welcome on
https://issues.apache.org/jira/browse/HDFS-1564 for achieving this!
On Wed, Mar 6, 2013 at 9:56 PM, John Meza <[EMAIL PROTECTED]> wrote:

> Thanks for the reply. This sounds like it has potential, but also seems to
> be a rather duct-tape type of work around. It would be nice if there was a
> mod to dfs.datanode.du.reserved that worked within Hadoop, so that would
> imply that hadoop was a little more certain to adhere to it.
>
> I understand that dfs.datanode.du.reserved defines reservd storage on each
> volume. I would like to give each volume a different reserved value.
>
> An example:
>
>  <name>dfs.datanode.du.reserved</name>
>    <value>///hstore1/dfs/dn:161061273600,///hstore2/dfs/dn:53687091200</value>
>
> Or something similiar.
> thanks
> John
> ------------------------------
> Date: Wed, 6 Mar 2013 10:25:17 +0100
> Subject: Re: dfs.datanode.du.reserved
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
>
> Not that I know. If so you should be able to identify each volume and as
> of now this isn't the case.
> BUT it can be done without Hadoop knowing about it, at the OS level, using
> different partitions/mounts for datanode and jobtracker stuff. That should
> solve your problem.
>
> Regards
>
> Bertrand
>
> On Mon, Mar 4, 2013 at 10:26 PM, John Meza <[EMAIL PROTECTED]> wrote:
>
> I'm probably not being clear.
> this seems to describe it: dfs.datanode.du.reserved  configured
> per-volume.
> https://issues.apache.org/jira/browse/HDFS-1564
>
> thanks
> John
> ------------------------------
> From: [EMAIL PROTECTED]
> Date: Mon, 4 Mar 2013 15:37:36 -0500
> Subject: Re: dfs.datanode.du.reserved
> To: [EMAIL PROTECTED]
>
>
> Possible to reserve 0 from various testing I have done yet that could
> cause the obvious side effect of achieving zero disk space:) Have only
> tested in development environment, however. Yet there are various tuning
> white papers and other benchmarks where the very same has been tested.
>
> Thanks.
>
> On Mon, Mar 4, 2013 at 2:00 PM, John Meza <[EMAIL PROTECTED]> wrote:
>
> the parameter: dfs.datanode.du.reserved is used to reserve disk space PER
> datanode. Is it possible to reserve a different amount of disk space per
> DISK?
>
> thanks
> John
>
>
>
>
> --
> Ellis R. Miller
> 937.829.2380
>
> <http://my.wisestamp.com/link?u=2hxhdfd4p76bkhcm&site=www.wisestamp.com/email-install>
>
> Mundo Nulla Fides
>
>
>
> <http://my.wisestamp.com/link?u=gfbmwhzrwxzcrjqx&site=www.wisestamp.com/email-install>
>
>
>
>
>
>
--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB