Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Re: [jira] [Commented] (FLUME-1667) Errors in source RegexExtractor


Copy link to this message
-
Re: [jira] [Commented] (FLUME-1667) Errors in source RegexExtractor
I have added a patch that is similar to the FlumeOG regex extractor
FLUME-1657 <https://issues.apache.org/jira/browse/FLUME-1657>

On Thu, Oct 25, 2012 at 10:37 AM, Sergey (JIRA) <[EMAIL PROTECTED]> wrote:

>
>     [
> https://issues.apache.org/jira/browse/FLUME-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484308#comment-13484308]
>
> Sergey commented on FLUME-1667:
> -------------------------------
>
> It seems that I can't try this on flume-ng because there's no
> RegexExtractor functionality now AFAIK. I use Regex extractor to compose a
> path within HDFS by using the first field of log record message body (date)
> and this is critical for me.
>
> > Errors in source RegexExtractor
> > -------------------------------
> >
> >                 Key: FLUME-1667
> >                 URL: https://issues.apache.org/jira/browse/FLUME-1667
> >             Project: Flume
> >          Issue Type: Bug
> >          Components: Sinks+Sources
> >    Affects Versions: v0.9.4
> >            Reporter: Sergey
> >
> > We are experiencing some problems with Flume that randomly happen on our
> systems 2 or more times per day. Here is what they look like...
> > This is the first sensible message in logs regarding this issue:
> > ...
> > 2012-10-23 15:02:27,021 INFO
> com.cloudera.flume.handlers.debug.StubbornAppendSink: append failed on
> event 'xservice-backend02 [INFO Tue Oct 23 15:02:12
> >  UTC 2012] { AckChecksum : (long)1613444596  (string) '....`+5�'
> (double)7.971475464E-315 } { AckTag :
> 20121023-150211682+0000.2380890497695726.00000068 } {
> > AckType : msg } { ds : 2012-10-23 } { tailSrcFile :
> operation.xservice-backend02.log } 2012-10-23 15:02:12,053
> \tRESP\txaxpxo\tf0bdc8efc9df943380359bc4
> > f5a1058e76\t/0/poster/\tPOST\t\"force_rewrite\": \"1\", \"session_key\":
> \"f0bdc8efc9df943380359bc4f5a1058e76\", \"filesize\": \"73\", \"name\":
> \"/Download/
> > aspect/TRANSITIVE AND ARGUMENT
> STRUCTURE.pdf\"\t6c86d15a5593c90381ef2d0b33fa7aa9\t0.0876088142395\t3\tpc:0.992:en:lan:Virtual
> Drive:\tCloud_Syncer\t' w
> > ith error: Blocked append interrupted by rotation event
> > After a few seconds exceptions appear:
> > ...
> > 2012-10-23 15:02:31,769 INFO
> com.cloudera.flume.handlers.debug.InsistentAppendDecorator: Failed due to
> unexpected runtime exception during append attempt
> > java.lang.RuntimeException: Blocked append interrupted by rotation event
> >         at
> com.cloudera.flume.handlers.rolling.RollSink.append(RollSink.java:215)
> >         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> >         at
> com.cloudera.flume.core.MaskDecorator.append(MaskDecorator.java:43)
> >         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> >         at
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.append(InsistentOpenDecorator.java:169)
> >         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> >         at
> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:71)
> >         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> >         at
> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
> >         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> >         at
> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:172)
> >         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> >         at
> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
> >         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> >         at
> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
> >         at
> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:241)

Thanks

Cameron Gandevia
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB