Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Exceptions after reloading configuration


+
Yatchmenoff, Sam 2013-01-16, 20:41
+
Brock Noland 2013-01-16, 21:46
+
Juhani Connolly 2013-01-17, 02:11
+
Brock Noland 2013-01-17, 02:18
+
Hari Shreedharan 2013-01-17, 02:28
+
Juhani Connolly 2013-01-17, 02:36
Copy link to this message
-
Re: Exceptions after reloading configuration
Agreed in 1.2 re-configuration was certainly buggy! :) and RollingFileSink doubly so.

--
Brock Noland
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Wednesday, January 16, 2013 at 6:36 PM, Juhani Connolly wrote:

> I hadn't noticed that patch, looks useful! Was mainly referring to 1.2
> where hot reconfiguration certainly wasn't reliable.
>
> So I guess Sam is looking at either upgrading(recommended since 1.3.1
> has lots of other goodness) and/or doing full restart on reconfiguration.
>
> On 01/17/2013 11:18 AM, Brock Noland wrote:
> > FLUME-1630 (in flume 1.3.1) hopefully improved re-configuration, but
> > in general I agree, restart is better.
> >
> > On Wed, Jan 16, 2013 at 6:11 PM, Juhani Connolly
> > <[EMAIL PROTECTED]> wrote:
> > > Switching configuration on a running node is pretty buggy, I would recommend
> > > just restarting flume, as while sometimes it will work there are issues like
> > > components not getting properly shut down even when removed from the config.
> > >
> > >
> > > On 01/17/2013 05:41 AM, Yatchmenoff, Sam wrote:
> > >
> > > I have Flume 1.2.0 running in a production system with 3 collectors fed by
> > > ~30 agents running on our application servers. If I make a change to the
> > > node configuration on the collectors, when the configuration is reloaded
> > > automatically, the collectors will occasionally fail and repeatedly report
> > > the following exception:
> > >
> > > 2013-01-16 20:31:22,353 ERROR flume.SinkRunner: Unable to deliver event.
> > > Exception follows.
> > > org.apache.flume.EventDeliveryException: Failed to process transaction
> > > at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:218)
> > > at
> > > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > at java.lang.Thread.run(Thread.java:679)
> > > Caused by: java.io.IOException: Stream Closed
> > > at java.io.FileOutputStream.writeBytes(Native Method)
> > > at java.io.FileOutputStream.write(FileOutputStream.java:297)
> > > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
> > > at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
> > > at
> > > org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71)
> > > at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:195)
> > > ... 3 more
> > >
> > > After about a dozen of those, I will start seeing this exception:
> > >
> > > 2013-01-16 20:32:27,374 ERROR flume.SinkRunner: Unable to deliver event.
> > > Exception follows.
> > > org.apache.flume.EventDeliveryException: Unable to rotate file
> > > /mnt/rawlog/1358365369665-49 while delivering event
> > > at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:155)
> > > at
> > > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > at java.lang.Thread.run(Thread.java:679)
> > > Caused by: java.io.IOException: Stream Closed
> > > at java.io.FileOutputStream.writeBytes(Native Method)
> > > at java.io.FileOutputStream.write(FileOutputStream.java:297)
> > > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> > > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> > > at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:149)
> > > ... 3 more
> > >
> > >
> > > Here is configuration for that agent:
> > >
> > > agent1.sources = source1 source2
> > > agent1.sinks = sink1
> > > agent1.channels = channel1
> > >
> > > # Describe/configure source1
> > > agent1.sources.source1.type = avro
> > > agent1.sources.source1.bind = 0.0.0.0
> > > agent1.sources.source1.port = 35853
> > >
> > > agent1.sources.source2.type = netcat
> > > agent1.sources.source2.bind = localhost
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB