Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> hdfs system crashes when loading files bigger than local space left


Copy link to this message
-
Re: hdfs system crashes when loading files bigger than local space left

On Jul 15, 2010, at 1:11 AM, Vitaliy Semochkin wrote:

> >a) Have you set a reserved size for hdfs?
> Yes. I set 128Mb as reserved size.

That is likely way too small.

> b) Are you loading data from the datanode?
> Yes. But the datanode is running on same node as namenode (i have very small cluster, only 5 servers and wasting one node only for namenode/jobtracker seemed unreasonable to me)

Where the NN is running is irrelevant to this particular problem.

The problem is that if you start your data load on a machine also running a datanode process, the data will get put onto that node first.  This will cause your DFS to be majorly unbalanced.

It is much better to load the data from another host outside the grid.