Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> HDFS DataNode daily log growing really high and fast


Copy link to this message
-
Re: HDFS DataNode daily log growing really high and fast
Hi,

On Mon, Oct 31, 2011 at 12:59 AM, Ronen Itkin <[EMAIL PROTECTED]> wrote:
> For instance, yesterday's daily log:
> "/var/log/hadoop/hadoop-hadoop-datanode-ip-10-10-10-4.log"
> on the problematic Node03 was in the size of 1.1 GB while on other Nodes
> the same log was in the size of 87 MB.
>
> Again, nothing is being run specifically on Node03, I have 3 nodes, with
> replication of 3 - means that all the data is being saved on every node,
> All nodes are connected to the same switch (and on the same subnet) - so no
> advantages to Node03 in any Job.
>
> I am being suspicious regarding HBase...
Does that servers regionserver have more regions assigned to it?
Check the HBase GUI.

Also, you can turn that message off with:

log4j.logger.org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace=WARN
Brock
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB