Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # user >> How to check whether chukwa collecting logs


+
Shouvanik Haldar 2013-01-09, 10:54
+
Shouvanik Haldar 2013-01-10, 07:11
Copy link to this message
-
Re: How to check whether chukwa collecting logs
Yes, the data is written to HDFS, but you probably need to run demux job to
organize the data into the form that is suitable for you.

regards,
Eric

On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <[EMAIL PROTECTED]
> wrote:

> I am trying to find out a check point , so that I can say that Chukwa is
> running and collecting logs
>
>
> I configured the /chukwa/etc/chukwa/initial_adaptors
> with following
>
> add FileTailer
> /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log
> 0
>
> .
> then I started bin/chukwa agent.
>
> Started the hadoop daemons.
>
> then, bin/chukwa collector.
>
> Next, I check the site
> "http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true"
>
> with following statistics
>
> Date:1357728723103
> Now:1357728765604
> numberHTTPConnection in time window:17
> numberchunks in time window:119
> lifetimechunks:23029
>
>
> and when I check hdfs I see
>
> -rw-r--r--   1 ec2-user supergroup     657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done
> -rw-r--r--   1 ec2-user supergroup     483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done
>
> -rw-r--r--   1 ec2-user supergroup     483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done
> -rw-r--r--   1 ec2-user supergroup     463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done
>
> -rw-r--r--   1 ec2-user supergroup     386337 2013-01-08 11:36 /chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done
> -rw-r--r--   1 ec2-user supergroup     479656 2013-01-08 11:41 /chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done
>
> -rw-r--r--   1 ec2-user supergroup     487748 2013-01-08 11:46 /chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done
> -rw-r--r--   1 ec2-user supergroup     479696 2013-01-08 11:51 /chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done
>
> -rw-r--r--   1 ec2-user supergroup     488030 2013-01-08 11:56 /chukwa/logs/201308115135034_domU12313907193B_488afeb413c19ee05bf7ff5.done
> -rw-r--r--   1 ec2-user supergroup     479754 2013-01-08 12:01 /chukwa/logs/201308115635034_domU12313907193B_488afeb413c19ee05bf7ff2.done
>
> -rw-r--r--   1 ec2-user supergroup     488067 2013-01-08 12:06 /chukwa/logs/201308120135034_domU12313907193B_488afeb413c19ee05bf7fef.done
> -rw-r--r--   1 ec2-user supergroup     479961 2013-01-08 12:11 /chukwa/logs/201308120635034_domU12313907193B_488afeb413c19ee05bf7fec.done
>
> -rw-r--r--   1 ec2-user supergroup     487976 2013-01-08 12:16 /chukwa/logs/201308121135034_domU12313907193B_488afeb413c19ee05bf7fe9.done
> -rw-r--r--   1 ec2-user supergroup     479863 2013-01-08 12:21 /chukwa/logs/201308121635034_domU12313907193B_488afeb413c19ee05bf7fe6.done
>
> -rw-r--r--   1 ec2-user supergroup     649168 2013-01-08 12:42 /chukwa/logs/201308123755390_domU12313907193B_5ab1511313c1a2ac24f8000.done
> -rw-r--r--   1 ec2-user supergroup     483972 2013-01-08 12:47 /chukwa/logs/201308124255466_domU12313907193B_5ab1511313c1a2ac24f7ffc.done
>
> -rw-r--r--   1 ec2-user supergroup     483417 2013-01-08 12:52 /chukwa/logs/201308124755466_domU12313907193B_5ab1511313c1a2ac24f7ff8.done
> -rw-r--r--   1 ec2-user supergroup     576711 2013-01-08 13:06 /chukwa/logs/201308130134839_domU12313907193B_2abeddbf13c1a406b0b8000.done
>
> -rw-r--r--   1 ec2-user supergroup     483877 2013-01-08 13:11 /chukwa/logs/201308130634912_domU12313907193B_2abeddbf13c1a406b0b7ffc.done
> -rw-r--r--   1 ec2-user supergroup     484352 2013-01-08 13:16 /chukwa/logs/201308131134912_domU12313907193B_2abeddbf13c1a406b0b7ff7.done
>
> -rw-r--r--   1 ec2-user supergroup     486158 2013-01-08 13:21 /chukwa/logs/201308131634912_domU12313907193B_2abeddbf13c1a406b0b7ff2.done
> -rw-r--r--   1 ec2-user supergroup     483925 2013-01-08 13:26 /chukwa/logs/201308132134913_domU12313907193B_2abeddbf13c1a406b0b7fed.done
+
Shouvanik Haldar 2013-01-10, 07:01
+
shreyas subramanya 2013-01-10, 07:24
+
Shouvanik Haldar 2013-01-10, 07:34
+
shreyas subramanya 2013-01-10, 07:44
+
Shouvanik Haldar 2013-01-10, 08:06
+
Shouvanik Haldar 2013-01-10, 09:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB