Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - hbase as sink


Copy link to this message
-
RE: hbase as sink
Chhaya Vishwakarma 2013-12-27, 03:10
Hi

host1.sources = src1
host1.sinks = sink1
host1.channels = ch1

host1.sources.src1.type = exec
host1.sources.src1.command = tail -F /home/user/Flume/flume-ng/csa-node1.log
host1.sources.src1.channels = ch1
host1.sources.src1.restart = true

host1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
host1.sinks.sink1.channel = ch1
host1.sinks.sink1.table = test10
host1.sinks.sink1.columnFamily = testing

host1.sinks.sink1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
host1.sinks.sink1.serializer.regex =(\.\w+)(?<=Exception)
host1.sinks.sink1.serializer.regexIgnoreCase = true
host1.sinks.sink1.serializer.colNames = exception

host1.channels.ch1.type=memory
Regards,
Chhaya Vishwakarma
From: Brock Noland [mailto:[EMAIL PROTECTED]]
Sent: Thursday, December 26, 2013 9:41 PM
To: [EMAIL PROTECTED]
Subject: Re: hbase as sink

Please share your sink configuration.

On Thu, Dec 26, 2013 at 4:05 AM, Chhaya Vishwakarma <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi,

I want to parse a log file to get the highlighted text.

Regular expression used to parse

(\.\w+)(?<=Exception)

But table in hbase shows no rows
The above Regex  is working on the below sample data in regex editor (gskinner)

13 Aug 2013 11:28:30,414 [WebContainer : 6] ERROR - An Error has occured for com.marsh.framework.core.exception.MarshException: Your session has timed out.
13 Aug 2013 11:28:30,414 [WebContainer : 6] ERROR - handleException():com.marsh.framework.core.exception.MarshException: Your session has timed out.
13 Aug 2013 15:45:12,061 [WebContainer : 2] ERROR - Remote Exception while updating CSA Details java.rmi.ServerException: RemoteException occurred in server thread;nested exception is:

What notation does flume follows for regex? (for e.g pearl notation)

Regards,
Chhaya Vishwakarma
________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"

--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org