Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - HDFS read/write data throttling


Copy link to this message
-
HDFS read/write data throttling
lohit 2013-11-11, 18:59
Hello Devs,

Wanted to reach out and see if anyone has thought about ability to throttle
data transfer within HDFS. One option we have been thinking is to throttle
on a per FileSystem basis, similar to Statistics in FileSystem. This would
mean anyone with handle to HDFS/Hftp will be throttled globally within JVM.
Right value to come up for this would be based on type of hardware we use
and how many tasks/clients we allow.

On the other hand doing something like this at FileSystem layer would mean
many other tasks such as Job jar copy, DistributedCache copy and any hidden
data movement would also be throttled. We wanted to know if anyone has had
such requirement on their clusters in the past and what was the thinking
around it. Appreciate your inputs/comments

--
Have a Nice Day!
Lohit