Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> hdfs system crashes when loading files bigger than local space left


+
Vitaliy Semochkin 2010-07-14, 11:16
+
Allen Wittenauer 2010-07-14, 15:23
+
Vitaliy Semochkin 2010-07-15, 08:11
Copy link to this message
-
Re: hdfs system crashes when loading files bigger than local space left

On Jul 15, 2010, at 1:11 AM, Vitaliy Semochkin wrote:

> >a) Have you set a reserved size for hdfs?
> Yes. I set 128Mb as reserved size.

That is likely way too small.

> b) Are you loading data from the datanode?
> Yes. But the datanode is running on same node as namenode (i have very small cluster, only 5 servers and wasting one node only for namenode/jobtracker seemed unreasonable to me)

Where the NN is running is irrelevant to this particular problem.

The problem is that if you start your data load on a machine also running a datanode process, the data will get put onto that node first.  This will cause your DFS to be majorly unbalanced.

It is much better to load the data from another host outside the grid.
+
Vitaliy Semochkin 2010-07-16, 10:15
+
Allen Wittenauer 2010-07-16, 18:07
+
Vitaliy Semochkin 2010-07-21, 10:02