Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> flume write ahead log is finished


Copy link to this message
-
Re: flume write ahead log is finished
sorry, I should've mentioned this at the beginning, we are running
flume 0.9.4-cdh3u4. It's not 1.3 unfortunately.

Shengjie
On 4 December 2012 21:08, Brock Noland <[EMAIL PROTECTED]> wrote:

> You probably want the JSon reporting:
> http://flume.apache.org/FlumeUserGuide.html#monitoring
>
> Brock
>
> On Tue, Dec 4, 2012 at 3:02 PM, Shengjie Min <[EMAIL PROTECTED]> wrote:
> >>Generally you should be fine to bring down the file channel at any
> >>point. That is why we have the WAL.
> >
> > I understand WAL is there in case you restart your flume agent. But in my
> > case I am switching from one flume agent to another type, let's say from
> > tailsource to AMQP agent. So before I shutdown tailsource I really want
> to
> > make sure WAL is all finished. How do you export the channel size in this
> > case?
> >
> >
> > On 4 December 2012 17:36, Brock Noland <[EMAIL PROTECTED]> wrote:
> >>
> >> Generally you should be fine to bring down the file channel at any
> >> point. That is why we have the WAL.
> >>
> >> If you wanted to be sure the channel was empty, we do export the
> >> channel size (depth).
> >>
> >> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hi,
> >> >
> >> > When it comes to maintenance outage, flume needs to be shutdown and
> >> > offline
> >> > for a while. We want to make sure flume's WAL is completed before we
> >> > shut
> >> > down the process. Is there anyway or any metrics exposed can indicate
> >> > WAL is
> >> > fully processed so that it's safe to bring it down gracefully?
> >> >
> >> > Of coz, the assumption here is that, the source of flume is not
> writing
> >> > any
> >> > new data, flume only has a certain size backlog needs to be cleaned
> up.
> >> >
> >> >
> >> > --
> >> > All the best,
> >> > Shengjie Min
> >> >
> >>
> >>
> >>
> >> --
> >> Apache MRUnit - Unit testing MapReduce -
> >> http://incubator.apache.org/mrunit/
> >
> >
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

--
All the best,
Shengjie Min
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB