Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> Writing logs collected by chukwa to hbase


Copy link to this message
-
Re: Writing logs collected by chukwa to hbase
I am not able to see any thing on hdfs:
when I browse using the url ->
http://ec2-54-234-57-44.compute-1.amazonaws.com:50070/nn_browsedfscontent.jsp
I see nothing..but

" Network Error (tcp_error)

  A communication error occurred: "Operation timed out"   The Web Server
may be down, too busy, or experiencing other problems preventing it from
responding to requests. You may wish to try again at a later time.
For assistance, contact your network support team. "
On Tue, Jan 8, 2013 at 6:22 PM, Shouvanik Haldar <[EMAIL PROTECTED]
> wrote:

> How I was sure data getting collected at collector, is by seeing the below
> statistics
>
>
> Date:1357649516006
> Now:1357649518803
> numberHTTPConnection in time window:2
> numberchunks in time window:7
> lifetimechunks:2612
>
>
>
>
> On Tue, Jan 8, 2013 at 6:20 PM, Shouvanik Haldar <
> [EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I was following the post -> http://www.undercloud.org/?p=429. I did the
>> steps as was mentioned here. But, when I was going to check in HDFS whether
>> data has come from chukwa, I cannot see the page. I get the exception as
>>
>>
>>
>> Also, I want to know what is the default location of chukwa logs. I found
>> out a location ->
>> "/home/ec2-user/chukwa/chukwa-incubating-0.5.0/sbin/logs". Is this the
>> location? Previously, I had some logs inside this directory. I have deleted
>> these logs. Now, no logs are getting created.
>>
>> Previously, when logs were existing, I checked one log entry. It says
>> "nice: /home/ec2-user/chukwa/chukwa-incubating-0.5.0/sbin/bin/chukwa: No
>> such file or directory".
>>
>>
>> I have edited bash_profile to give the path of CHUKWA_HOME. Is this
>> required?
>>
>> I am using "chukwa-incubating-0.5.0" and "hadoop-1.0.4". I can see data
>> getting collected at the collector by browsing the url =>
>> http://ec2-54-234-57-44.compute-1.amazonaws.com:8080/chukwa?ping=true
>>
>>
>>
>>
>>
>> But somehow, data not getting inside hdfs. Please help.
>>
>>
>> Regards,
>> Shouvanik
>>
>>
>> On Tue, Jan 8, 2013 at 10:22 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>
>>> The quick start guide may help:
>>>
>>> http://incubator.apache.org/chukwa/docs/r0.5.0/Quick_Start_Guide.html
>>>
>>> regards,
>>> Eric
>>>
>>>
>>> On Mon, Jan 7, 2013 at 1:17 AM, Shouvanik Haldar <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>> I want to install chukwa on a single node and test it. Where can I find
>>>> a suitable document?
>>>>
>>>>
>>>> On Mon, Jan 7, 2013 at 2:47 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> There are a few things that you need to do:
>>>>>
>>>>> 1. Configure the collector to use HBaseWriter.
>>>>>
>>>>>   http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html
>>>>>
>>>>> Make sure that hadoop and hbase configuration are in the classpath of
>>>>> the collector.
>>>>>
>>>>> 2. Write a demux parser for your log file to store on HBase, and this
>>>>> runs in chukwa collector.
>>>>>
>>>>>   http://incubator.apache.org/chukwa/docs/r0.5.0/programming.html
>>>>>
>>>>> 3. What log4j adaptors and appenders are you using to write the log
>>>>> files?
>>>>>   I would recommend to use socket appender from Log4j and use socket
>>>>> adaptor to listen for the incoming data stream.  This will ensure the
>>>>> records are delimited properly.
>>>>>
>>>>> Hope this helps.
>>>>>
>>>>> regards,
>>>>> Eric
>>>>>
>>>>> On Sun, Jan 6, 2013 at 6:00 AM, NehaS Singh <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> I am using chukwa 0.5. My probleam is want to store the logs from
>>>>>> log4j or other party which is not hadoop.Also i want to  write the logs to
>>>>>> hbase.
>>>>>> But i am not ale to do so. I have configuerd log4j.properties file so
>>>>>> i am able to see th logs in hdfs.
>>>>>> what are the configuration or steps using which i can write the logs
>>>>>> to hbase.
>>>>>> Please help.
>>>>>> ________________________________________
>>>>>> From: Eric Yang [[EMAIL PROTECTED]]
>>>>>> Sent: Saturday, January 05, 2013 11:55 PM

Thanks,
*Shouvanik*