Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa, mail # user - Writing logs collected by chukwa to hbase


Copy link to this message
-
Re: Writing logs collected by chukwa to hbase
Eric Yang 2013-01-06, 21:17
There are a few things that you need to do:

1. Configure the collector to use HBaseWriter.

  http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html

Make sure that hadoop and hbase configuration are in the classpath of the
collector.

2. Write a demux parser for your log file to store on HBase, and this runs
in chukwa collector.

  http://incubator.apache.org/chukwa/docs/r0.5.0/programming.html

3. What log4j adaptors and appenders are you using to write the log files?
  I would recommend to use socket appender from Log4j and use socket
adaptor to listen for the incoming data stream.  This will ensure the
records are delimited properly.

Hope this helps.

regards,
Eric

On Sun, Jan 6, 2013 at 6:00 AM, NehaS Singh <[EMAIL PROTECTED]>wrote:

> I am using chukwa 0.5. My probleam is want to store the logs from log4j or
> other party which is not hadoop.Also i want to  write the logs to hbase.
> But i am not ale to do so. I have configuerd log4j.properties file so i am
> able to see th logs in hdfs.
> what are the configuration or steps using which i can write the logs to
> hbase.
> Please help.
> ________________________________________
> From: Eric Yang [[EMAIL PROTECTED]]
> Sent: Saturday, January 05, 2013 11:55 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Writing logs collected by chukwa to hbase
>
> What version of Chukwa are you running?
>
> regards,
> Eric
>
> On Fri, Dec 28, 2012 at 2:56 AM, NehaS Singh <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
> Hi,
>                 I am using Chukwa to collect log4j logs which I can see in
> HDFS. But I am not able to write it to HBASE. Can anyone help ?
>
> Regards,
> Neha Singh
>
>
> ________________________________
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>
>