Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - How to configure Regex in file names flume 1.3


Copy link to this message
-
Re: How to configure Regex in file names flume 1.3
Roshan Naik 2013-07-26, 00:36
you will need to build a custom version...

1) checkout flume-1.3  sources..

2) apply this patch..

https://issues.apache.org/jira/secure/attachment/12572794/1661.patch.v9

3) build.

-roshan

On Tue, Jul 23, 2013 at 11:54 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:

>  Hi Roshan,****
>
> I am using flume 1.3. Don’t we have a way to support regex in 1.3?****
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> ** **
>
> ** **
>
> *From:* Roshan Naik [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, July 23, 2013 3:10 PM
> *To:* [EMAIL PROTECTED]
> *Cc:* Kumar, Deepak8 [CCC-OT_IT NE]; [EMAIL PROTECTED];
> [EMAIL PROTECTED]
>
> *Subject:* Re: How to configure Regex in file names flume 1.3****
>
>  ** **
>
> Deepak,****
>
>    which version of flume are u using ? regex support is in 1.4, but you
> will need to use the 'shell' configuration in Exec Source for that.. see
> the User Guide for an example****
>
> ** **
>
> -roshan****
>
> ** **
>
> On Tue, Jul 23, 2013 at 10:50 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>
> wrote:****
>
> Could anyone assist on this?****
>
>  ****
>
> Could we have a regex in tail –F****
>
>  ****
>
> Regards,****
>
> Deepak****
>
>  ****
>
> *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> *Sent:* Sunday, July 21, 2013 12:29 PM
> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> [EMAIL PROTECTED]'****
>
>
> *Subject:* RE: How to configure Regex in file names flume 1.3****
>
>  ****
>
> A simple tail works if I give the exact file name. But if I give a regex
> as file name like****
>
> agent.sources.source1.command = tail -F /var/log/flume-ng/template.*****
>
>  ****
>
> then it is not working.****
>
>  ****
>
> Regards,****
>
> Deepak****
>
>  ****
>
> *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> *Sent:* Sunday, July 21, 2013 12:27 PM
> *To:* '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Kumar, Deepak8
> [CCC-OT_IT NE]
> *Subject:* RE: How to configure Regex in file names flume 1.3****
>
>  ****
>
> Hi,****
>
> I have checked it but it is not working. I verified from JConsole but it
> is not sending the log events.****
>
>  ****
>
> Do we have any other way to cross check?****
>
>  ****
>
> Regards,****
>
> Deepak****
>
>  ****
>
> *From:* Israel Ekpo [mailto:[EMAIL PROTECTED] <[EMAIL PROTECTED]>]
> *Sent:* Saturday, July 20, 2013 4:46 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: How to configure Regex in file names flume 1.3****
>
>  ****
>
> Deepak,****
>
>  ****
>
> The sample configuration you have looks valid****
>
>  ****
>
> Try it out and let us know if you are experiencing any problems****
>
>  ****
>
> More information on how to configure the Exec Source is available here.***
> *
>
>  ****
>
> http://flume.apache.org/FlumeUserGuide.html#exec-source****
>
>  ****
>
>
> ****
>
> *Author and Instructor for the Upcoming Book and Lecture Series*****
>
> *Massive Log Data Aggregation, Processing, Searching and Visualization
> with Open Source Software*****
>
> *http://massivelogdata.com*****
>
>  ****
>
> On 19 July 2013 14:00, Kumar, Deepak8 <[EMAIL PROTECTED]> wrote:****
>
> Hi,****
>
> Could anyone please assist on this?****
>
>  ****
>
> Regards,****
>
> Deepak****
>
>  ****
>
> *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> *Sent:* Monday, July 15, 2013 8:00 AM
> *To:* '[EMAIL PROTECTED]'
> *Cc:* Kumar, Deepak8 [CCC-OT_IT NE]
> *Subject:* How to configure Regex in file names flume 1.3****
>
>  ****
>
> Hi,****
>
> I am using flume 1.3. Could we configure files names as regular
> expressions in flume-conf.properties.****
>
>  ****
>
> e.g.****
>
> agent.sources.source1.type = exec****
>
> agent.sources.source1.command = tail -F /var/log/flume-ng/template.log****
>
> agent.sources.source1.channels = fileChannel****
>
> agent.sources.source1.interceptors = logIntercept****
>
>  ****
>
> Could we have it something like this in flume 1.3****
>
> agent.sources.source1.command = tail -F /var/log/flume-ng/template.*****
>
>  ****
>
> Regards,****
>
> Deepak****
>
>  ****
>
> ** **