Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # user >> Writing logs collected by chukwa to hbase


+
NehaS Singh 2012-12-28, 10:56
+
Eric Yang 2013-01-05, 18:25
+
NehaS Singh 2013-01-06, 14:00
+
Eric Yang 2013-01-06, 21:17
+
Shouvanik Haldar 2013-01-07, 09:17
+
Eric Yang 2013-01-08, 04:52
+
Shouvanik Haldar 2013-01-08, 12:50
Copy link to this message
-
Re: Writing logs collected by chukwa to hbase
How I was sure data getting collected at collector, is by seeing the below
statistics
Date:1357649516006
Now:1357649518803
numberHTTPConnection in time window:2
numberchunks in time window:7
lifetimechunks:2612
On Tue, Jan 8, 2013 at 6:20 PM, Shouvanik Haldar <[EMAIL PROTECTED]
> wrote:

> Hi,
>
> I was following the post -> http://www.undercloud.org/?p=429. I did the
> steps as was mentioned here. But, when I was going to check in HDFS whether
> data has come from chukwa, I cannot see the page. I get the exception as
>
>
>
> Also, I want to know what is the default location of chukwa logs. I found
> out a location ->
> "/home/ec2-user/chukwa/chukwa-incubating-0.5.0/sbin/logs". Is this the
> location? Previously, I had some logs inside this directory. I have deleted
> these logs. Now, no logs are getting created.
>
> Previously, when logs were existing, I checked one log entry. It says
> "nice: /home/ec2-user/chukwa/chukwa-incubating-0.5.0/sbin/bin/chukwa: No
> such file or directory".
>
>
> I have edited bash_profile to give the path of CHUKWA_HOME. Is this
> required?
>
> I am using "chukwa-incubating-0.5.0" and "hadoop-1.0.4". I can see data
> getting collected at the collector by browsing the url =>
> http://ec2-54-234-57-44.compute-1.amazonaws.com:8080/chukwa?ping=true
>
>
>
>
>
> But somehow, data not getting inside hdfs. Please help.
>
>
> Regards,
> Shouvanik
>
>
> On Tue, Jan 8, 2013 at 10:22 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> The quick start guide may help:
>>
>> http://incubator.apache.org/chukwa/docs/r0.5.0/Quick_Start_Guide.html
>>
>> regards,
>> Eric
>>
>>
>> On Mon, Jan 7, 2013 at 1:17 AM, Shouvanik Haldar <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>> I want to install chukwa on a single node and test it. Where can I find
>>> a suitable document?
>>>
>>>
>>> On Mon, Jan 7, 2013 at 2:47 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>>
>>>> There are a few things that you need to do:
>>>>
>>>> 1. Configure the collector to use HBaseWriter.
>>>>
>>>>   http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html
>>>>
>>>> Make sure that hadoop and hbase configuration are in the classpath of
>>>> the collector.
>>>>
>>>> 2. Write a demux parser for your log file to store on HBase, and this
>>>> runs in chukwa collector.
>>>>
>>>>   http://incubator.apache.org/chukwa/docs/r0.5.0/programming.html
>>>>
>>>> 3. What log4j adaptors and appenders are you using to write the log
>>>> files?
>>>>   I would recommend to use socket appender from Log4j and use socket
>>>> adaptor to listen for the incoming data stream.  This will ensure the
>>>> records are delimited properly.
>>>>
>>>> Hope this helps.
>>>>
>>>> regards,
>>>> Eric
>>>>
>>>> On Sun, Jan 6, 2013 at 6:00 AM, NehaS Singh <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> I am using chukwa 0.5. My probleam is want to store the logs from
>>>>> log4j or other party which is not hadoop.Also i want to  write the logs to
>>>>> hbase.
>>>>> But i am not ale to do so. I have configuerd log4j.properties file so
>>>>> i am able to see th logs in hdfs.
>>>>> what are the configuration or steps using which i can write the logs
>>>>> to hbase.
>>>>> Please help.
>>>>> ________________________________________
>>>>> From: Eric Yang [[EMAIL PROTECTED]]
>>>>> Sent: Saturday, January 05, 2013 11:55 PM
>>>>> To: [EMAIL PROTECTED]
>>>>> Subject: Re: Writing logs collected by chukwa to hbase
>>>>>
>>>>> What version of Chukwa are you running?
>>>>>
>>>>> regards,
>>>>> Eric
>>>>>
>>>>> On Fri, Dec 28, 2012 at 2:56 AM, NehaS Singh <
>>>>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>>>>> wrote:
>>>>> Hi,
>>>>>                 I am using Chukwa to collect log4j logs which I can
>>>>> see in HDFS. But I am not able to write it to HBASE. Can anyone help ?
>>>>>
>>>>> Regards,
>>>>> Neha Singh
>>>>>
>>>>>
>>>>> ________________________________
>>>>> The contents of this e-mail and any attachment(s) may contain
>>>>> confidential or privileged information for the intended recipient(s).

Thanks,
*Shouvanik*
+
Shouvanik Haldar 2013-01-08, 13:04
+
Shouvanik Haldar 2013-01-08, 14:50
+
NehaS Singh 2013-01-07, 06:18
+
Eric Yang 2013-01-08, 04:51
+
Sreepathi 2012-12-28, 19:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB