Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Event breaking in flume


+
Chhaya Vishwakarma 2013-12-30, 09:53
+
Ashish 2013-12-30, 10:17
+
Chhaya Vishwakarma 2013-12-30, 10:26
+
Ashish 2013-12-30, 10:34
+
Joao Salcedo 2013-12-30, 10:51
+
Chhaya Vishwakarma 2013-12-30, 10:56
+
Joao Salcedo 2013-12-30, 11:05
+
Brock Noland 2013-12-30, 14:17
+
Chhaya Vishwakarma 2013-12-31, 03:54
Copy link to this message
-
Re: Event breaking in flume
Have a look at org.apache.flume.serialization.LineDeserializer in
flume-ng-core module
On Tue, Dec 31, 2013 at 9:24 AM, Chhaya Vishwakarma <
[EMAIL PROTECTED]> wrote:

> Hi brock
>
>
>
> Thanks. Using spooling directory with deserializer looks good however i
> don’t have any idea of how to write custom deserializer.
>
> Can you give me little hint how should i go about writing my own
> deserializer it will be a great help.
>
>
>
>
>
> Regards,
>
> Chhaya Vishwakarma
>
>
>
>
>
>
>
> *From:* Brock Noland [mailto:[EMAIL PROTECTED]]
> *Sent:* Monday, December 30, 2013 7:48 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Event breaking in flume
>
>
>
> Yes, it is possible to handle multi-line events and handling stack traces
> is very common place.
>
>
>
> However, using exec source is going to be limiting. The "correct" solution
> is:
>
>
>
> 1) Use spooling directory source
>
> 2) Write a little deserializer to handle your format.
>
>
>
> Another solution is:
>
>
>
> 1) replace new lines with something like __NL__ by a perl script in your
> exec source
>
> 2) Use morphlines to replace __NL__ with \n
>
>
>
> A third and less desirable solution would be:
>
>
>
> 1) Use the morphlines intercepter to merge multiple events to a single
> event. This will not work well for a varity or reasons but the most common
> being that the exec source could hit it's "batch" size in the middle of of
> a stack trace in which case the stack trace will be in to different batches.
>
>
>
> Brock
>
>
>
>
>
> On Mon, Dec 30, 2013 at 5:05 AM, Joao Salcedo <[EMAIL PROTECTED]>
> wrote:
>
> Looks that it is possible based on regular expression pattern matching
>
>
>
>
> http://kitesdk.org/docs/current/kite-morphlines/morphlinesReferenceGuide.html#/readMultiLine
>
>
>
> On Mon, Dec 30, 2013 at 9:56 PM, Chhaya Vishwakarma <
> [EMAIL PROTECTED]> wrote:
>
> So is it not possible to handle multiline events in flume?
>
>
>
> *From:* Joao Salcedo [mailto:[EMAIL PROTECTED]]
> *Sent:* Monday, December 30, 2013 4:22 PM
>
>
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Event breaking in flume
>
>
>
> Maybe you can set up some morphlines and do some ETL in your event.
>
>
>
> I hope this help you.
>
>
>
>
> http://blog.cloudera.com/blog/2013/07/morphlines-the-easy-way-to-build-and-integrate-etl-apps-for-apache-hadoop/
>
>
>
> Cheers
>
>
>
> On Mon, Dec 30, 2013 at 9:34 PM, Ashish <[EMAIL PROTECTED]> wrote:
>
> I am not aware of any options out of the box. Maybe someone else can help.
>
> Alternate way is to write a custom source.
>
>
>
> On Mon, Dec 30, 2013 at 3:56 PM, Chhaya Vishwakarma <
> [EMAIL PROTECTED]> wrote:
>
> Hi
>
> Exec as source and tail command
>
>
>
>
>
> *From:* Ashish [mailto:[EMAIL PROTECTED]]
> *Sent:* Monday, December 30, 2013 3:48 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Event breaking in flume
>
>
>
> What is the Source you are using?
>
>
>
> On Mon, Dec 30, 2013 at 3:23 PM, Chhaya Vishwakarma <
> [EMAIL PROTECTED]> wrote:
>
> Hi,
>
>
>
> By default flume considers one line as one event, But I want to do
> breaking on some other criteria how it can be achieved in flume? Is it
> possible to do ?
>
>
>
> 10 Sep 2013 19:43:33,561 [WebContainer : 9] ERROR - An Error has occured
> for com.marsh.framework.core.exception.MarshException: Record has been
> modified since last retrieved - Resubmit transaction
>
>
>
> 10 Sep 2013 19:43:33,561 [WebContainer : 9] ERROR -
> handleException():com.marsh.framework.core.exception.MarshException: Record
> has been modified since last retrieved - Resubmit transaction
>
>      at
> com.marsh.csa.serviceagreement.ServiceAgreementImpl.updateAgreement(ServiceAgreementImpl.java(Compiled
> Code))
>
>      at
> com.marsh.csa.serviceagreementmgmt.CSAManagerImpl.updateCSA(CSAManagerImpl.java(Compiled
> Code))
>
>      at
> com.marsh.csa.serviceagreementmgmt.ejb.EJSRemoteStatelessServiceagreementManager_3dcfd156.updateCSA(Unknown

thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
+
Chhaya Vishwakarma 2013-12-31, 06:49
+
Brock Noland 2013-12-31, 14:54
+
Chhaya Vishwakarma 2014-01-02, 05:40
+
Christopher Shannon 2013-12-30, 16:48
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB