Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - HDFS read/write data throttling


Copy link to this message
-
Re: HDFS read/write data throttling
lohit 2013-11-11, 19:47
Hi Adam,

Thanks for the reply. The changes I was referring was in FileSystem.java
layer which should not affect HDFS Replication/NameNode operations.
To give better idea this would affect clients something like this

Configuration conf = new Configuration();
conf.setInt("read.bandwitdh.mbpersec", 20); // 20MB/s
FileSystem fs = FileSystem.get(conf);

FSDataInputStream fis = fs.open("/path/to/file.xt");
fis.read(); // <-- This would be max of 20MB/s
2013/11/11 Adam Muise <[EMAIL PROTECTED]>

> See https://issues.apache.org/jira/browse/HDFS-3475
>
> Please note that this has met with many unexpected impacts on workload. Be
> careful and be mindful of your Datanode memory and network capacity.
>
>
>
>
> On Mon, Nov 11, 2013 at 1:59 PM, lohit <[EMAIL PROTECTED]> wrote:
>
> > Hello Devs,
> >
> > Wanted to reach out and see if anyone has thought about ability to
> throttle
> > data transfer within HDFS. One option we have been thinking is to
> throttle
> > on a per FileSystem basis, similar to Statistics in FileSystem. This
> would
> > mean anyone with handle to HDFS/Hftp will be throttled globally within
> JVM.
> > Right value to come up for this would be based on type of hardware we use
> > and how many tasks/clients we allow.
> >
> > On the other hand doing something like this at FileSystem layer would
> mean
> > many other tasks such as Job jar copy, DistributedCache copy and any
> hidden
> > data movement would also be throttled. We wanted to know if anyone has
> had
> > such requirement on their clusters in the past and what was the thinking
> > around it. Appreciate your inputs/comments
> >
> > --
> > Have a Nice Day!
> > Lohit
> >
>
>
>
> --
>    * Adam Muise *       Solutions Engineer
> ------------------------------
>
>     Phone:        416-417-4037
>   Email:      [EMAIL PROTECTED]
>   Website:   http://www.hortonworks.com/
>
>       * Follow Us: *
> <
> http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature
> >
> <
> http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature
> >
> <
> http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature
> >
>
>  [image: photo]
>
>   Latest From Our Blog:  How to use R and other non-Java languages in
> MapReduce and Hive
> <
> http://hortonworks.com/blog/using-r-and-other-non-java-languages-in-mapreduce-and-hive/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

--
Have a Nice Day!
Lohit