Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> flume write ahead log is finished


Copy link to this message
-
Re: flume write ahead log is finished
>Generally you should be fine to bring down the file channel at any
>point. That is why we have the WAL.

I understand WAL is there in case you restart your flume agent. But in my
case I am switching from one flume agent to another type, let's say from
tailsource to AMQP agent. So before I shutdown tailsource I really want to
make sure WAL is all finished. How do you export the channel size in this
case?
On 4 December 2012 17:36, Brock Noland <[EMAIL PROTECTED]> wrote:

> Generally you should be fine to bring down the file channel at any
> point. That is why we have the WAL.
>
> If you wanted to be sure the channel was empty, we do export the
> channel size (depth).
>
> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > When it comes to maintenance outage, flume needs to be shutdown and
> offline
> > for a while. We want to make sure flume's WAL is completed before we shut
> > down the process. Is there anyway or any metrics exposed can indicate
> WAL is
> > fully processed so that it's safe to bring it down gracefully?
> >
> > Of coz, the assumption here is that, the source of flume is not writing
> any
> > new data, flume only has a certain size backlog needs to be cleaned up.
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

--
All the best,
Shengjie Min
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB