Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> HDFS read/write data throttling

Copy link to this message
HDFS read/write data throttling
Hello Devs,

Wanted to reach out and see if anyone has thought about ability to throttle
data transfer within HDFS. One option we have been thinking is to throttle
on a per FileSystem basis, similar to Statistics in FileSystem. This would
mean anyone with handle to HDFS/Hftp will be throttled globally within JVM.
Right value to come up for this would be based on type of hardware we use
and how many tasks/clients we allow.

On the other hand doing something like this at FileSystem layer would mean
many other tasks such as Job jar copy, DistributedCache copy and any hidden
data movement would also be throttled. We wanted to know if anyone has had
such requirement on their clusters in the past and what was the thinking
around it. Appreciate your inputs/comments

Have a Nice Day!
Adam Muise 2013-11-11, 19:27
lohit 2013-11-11, 19:47
Haosong Huang 2013-11-12, 02:16
Andrew Wang 2013-11-12, 03:44
lohit 2013-11-12, 04:24
Steve Loughran 2013-11-12, 09:38
Andrew Wang 2013-11-13, 06:27
Steve Loughran 2013-11-13, 10:54
Andrew Wang 2013-11-18, 18:25
Jay Vyas 2013-11-18, 18:46
Andrew Wang 2013-11-18, 21:25