Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> hdfs system crashes when loading files bigger than local space left


+
Vitaliy Semochkin 2010-07-14, 11:16
+
Allen Wittenauer 2010-07-14, 15:23
+
Vitaliy Semochkin 2010-07-15, 08:11
+
Allen Wittenauer 2010-07-15, 17:26
+
Vitaliy Semochkin 2010-07-16, 10:15
Copy link to this message
-
Re: hdfs system crashes when loading files bigger than local space left

On Jul 16, 2010, at 3:15 AM, Vitaliy Semochkin wrote:
> That is likely way too small.
> Will setting 512Mb be better in case the whole volume size is only 190Gb?

I'd recommend at least 5gb.  I'm also assuming this same disk space isn't getting used for MapReduce.

> Does hadoop detect/distinct the client that uploads data from datanode and not from datanode?
> lets say I execute

Yes.

> hadoop -put someFile hdfs://namenode.mycompany.com/
>
> from namenode.mycompany.com and from some other pc. Will it be any different for hadoop and will hadoop orgonize data more balanced in the last case?

Yes.

Again, namenode is irrelevant. Do not do put's from a datanode if you want the data to be reasonably balanced.
+
Vitaliy Semochkin 2010-07-21, 10:02
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB