Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> Writing logs collected by chukwa to hbase


Copy link to this message
-
Re: Writing logs collected by chukwa to hbase
How I was sure data getting collected at collector, is by seeing the below
statistics
Date:1357649516006
Now:1357649518803
numberHTTPConnection in time window:2
numberchunks in time window:7
lifetimechunks:2612
On Tue, Jan 8, 2013 at 6:20 PM, Shouvanik Haldar <[EMAIL PROTECTED]
> wrote:

> Hi,
>
> I was following the post -> http://www.undercloud.org/?p=429. I did the
> steps as was mentioned here. But, when I was going to check in HDFS whether
> data has come from chukwa, I cannot see the page. I get the exception as
>
>
>
> Also, I want to know what is the default location of chukwa logs. I found
> out a location ->
> "/home/ec2-user/chukwa/chukwa-incubating-0.5.0/sbin/logs". Is this the
> location? Previously, I had some logs inside this directory. I have deleted
> these logs. Now, no logs are getting created.
>
> Previously, when logs were existing, I checked one log entry. It says
> "nice: /home/ec2-user/chukwa/chukwa-incubating-0.5.0/sbin/bin/chukwa: No
> such file or directory".
>
>
> I have edited bash_profile to give the path of CHUKWA_HOME. Is this
> required?
>
> I am using "chukwa-incubating-0.5.0" and "hadoop-1.0.4". I can see data
> getting collected at the collector by browsing the url =>
> http://ec2-54-234-57-44.compute-1.amazonaws.com:8080/chukwa?ping=true
>
>
>
>
>
> But somehow, data not getting inside hdfs. Please help.
>
>
> Regards,
> Shouvanik
>
>
> On Tue, Jan 8, 2013 at 10:22 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> The quick start guide may help:
>>
>> http://incubator.apache.org/chukwa/docs/r0.5.0/Quick_Start_Guide.html
>>
>> regards,
>> Eric
>>
>>
>> On Mon, Jan 7, 2013 at 1:17 AM, Shouvanik Haldar <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>> I want to install chukwa on a single node and test it. Where can I find
>>> a suitable document?
>>>
>>>
>>> On Mon, Jan 7, 2013 at 2:47 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>>
>>>> There are a few things that you need to do:
>>>>
>>>> 1. Configure the collector to use HBaseWriter.
>>>>
>>>>   http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html
>>>>
>>>> Make sure that hadoop and hbase configuration are in the classpath of
>>>> the collector.
>>>>
>>>> 2. Write a demux parser for your log file to store on HBase, and this
>>>> runs in chukwa collector.
>>>>
>>>>   http://incubator.apache.org/chukwa/docs/r0.5.0/programming.html
>>>>
>>>> 3. What log4j adaptors and appenders are you using to write the log
>>>> files?
>>>>   I would recommend to use socket appender from Log4j and use socket
>>>> adaptor to listen for the incoming data stream.  This will ensure the
>>>> records are delimited properly.
>>>>
>>>> Hope this helps.
>>>>
>>>> regards,
>>>> Eric
>>>>
>>>> On Sun, Jan 6, 2013 at 6:00 AM, NehaS Singh <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> I am using chukwa 0.5. My probleam is want to store the logs from
>>>>> log4j or other party which is not hadoop.Also i want to  write the logs to
>>>>> hbase.
>>>>> But i am not ale to do so. I have configuerd log4j.properties file so
>>>>> i am able to see th logs in hdfs.
>>>>> what are the configuration or steps using which i can write the logs
>>>>> to hbase.
>>>>> Please help.
>>>>> ________________________________________
>>>>> From: Eric Yang [[EMAIL PROTECTED]]
>>>>> Sent: Saturday, January 05, 2013 11:55 PM
>>>>> To: [EMAIL PROTECTED]
>>>>> Subject: Re: Writing logs collected by chukwa to hbase
>>>>>
>>>>> What version of Chukwa are you running?
>>>>>
>>>>> regards,
>>>>> Eric
>>>>>
>>>>> On Fri, Dec 28, 2012 at 2:56 AM, NehaS Singh <
>>>>> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
>>>>> wrote:
>>>>> Hi,
>>>>>                 I am using Chukwa to collect log4j logs which I can
>>>>> see in HDFS. But I am not able to write it to HBASE. Can anyone help ?
>>>>>
>>>>> Regards,
>>>>> Neha Singh
>>>>>
>>>>>
>>>>> ________________________________
>>>>> The contents of this e-mail and any attachment(s) may contain
>>>>> confidential or privileged information for the intended recipient(s).

Thanks,
*Shouvanik*