My cluster started being incredibly slow in the past 2 days.
I've seen many Blocking updates on the region server logs, which lead me to believe HDFS creates is the bottleneck.
I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which copies a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it was previously around 28MB/sec.
I have 3 RS computers, hosting DataNode and embedded zookeeper as well.
And 1 Master computer, hosting NameNode as well.
The slowest computer has Core 2 Duo 2.93GHz, 4GB.
What steps can I take to find out the source of this slowdown?