Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - flume agent not starting


Copy link to this message
-
RE: flume agent not starting
Chhaya Vishwakarma 2014-01-22, 07:09
Hi,

Is my interceptor configuration correct? Or there is some problem with that ?
Can someone guide me in this
Instead of tail I am using CAT now but same error its giving

Regards,
Chhaya Vishwakarma

-----Original Message-----
From: Chhaya Vishwakarma [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, January 22, 2014 10:20 AM
To: [EMAIL PROTECTED]
Subject: RE: flume agent not starting

Hi ,

What i should use then for interceptor to work shall I use CAT ??

-----Original Message-----
From: Wolfgang Hoschek [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, January 21, 2014 6:30 PM
To: [EMAIL PROTECTED]
Subject: Re: flume agent not starting

'tail' with exec source spits out one flume event per line into the interceptor. But readMultiLine expects multiple lines per event, not one line per event. In other words, by the time the data arrives in the interceptor it's already too late for readMultiLine to make sense.

Wolfgang.
On Jan 17, 2014, at 2:42 PM, Chhaya Vishwakarma wrote:

> Hi,
>  
> Agent is starting I am using morphline interceptor to parse events
>  
> Below is my morphline configuration
>  
> morphlines : [
> {
> id : morphline1
>  
> commands : [
> {
> readMultiLine {
>   regex : "Started session*"
>   what : next
>   charset : UTF-8
> }
> }
> ]
> }
> ]
>  
> This is my flume configuration file
>  
> host1.sources = src1
> host1.sinks = sink1
> host1.channels = ch1
>  
> host1.sources.src1.type = exec
> host1.sources.src1.command = tail -F
> /home/user/apache-flume-1.4.0-bin/try.log
> host1.sources.src1.channels = ch1
> host1.sources.src1.restart = true
> #host1.sources.src1.batchsize=10
>  
> host1.sources.src1.interceptors = morphlineinterceptor
> host1.sources.src1.interceptors.morphlineinterceptor.type =
> org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
> host1.sources.src1.interceptors.morphlineinterceptor.morphlineFile =
> /home/user/apache-flume-1.4.0-bin/morphline.conf
>  
> host1.sinks.sink1.type = file_roll
> host1.sinks.sink1.channel = ch1
> host1.sinks.sink1.sink.directory = /home/user/apache-flume-1.4.0-bin
> host1.sinks.sink1.sink.serializer= TEXT
> host1.sinks.sink1.sink.rollInterval=0
>  
> host1.channels.ch1.capacity=1000
> host1.channels.ch1.transactionCapacity=100
>  
> host1.channels.ch1.type=memory
>  
> input file content
>  
> Started session
> a
> b
> c
> Started session
> c
> d
> e
> Started session
>  
> But in /home/user/apache-flume-1.4.0-bin I can see a file named
> 1389960784130-1 but it's not showing any content. I can see only new
> line characters in output file
>  
> Regards,
> Chhaya Vishwakarma
>  
>
> The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"