Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> flume write ahead log is finished


+
Shengjie Min 2012-12-04, 17:26
+
Brock Noland 2012-12-04, 17:36
+
Shengjie Min 2012-12-04, 21:02
+
Brock Noland 2012-12-04, 21:08
Copy link to this message
-
Re: flume write ahead log is finished
sorry, I should've mentioned this at the beginning, we are running
flume 0.9.4-cdh3u4. It's not 1.3 unfortunately.

Shengjie
On 4 December 2012 21:08, Brock Noland <[EMAIL PROTECTED]> wrote:

> You probably want the JSon reporting:
> http://flume.apache.org/FlumeUserGuide.html#monitoring
>
> Brock
>
> On Tue, Dec 4, 2012 at 3:02 PM, Shengjie Min <[EMAIL PROTECTED]> wrote:
> >>Generally you should be fine to bring down the file channel at any
> >>point. That is why we have the WAL.
> >
> > I understand WAL is there in case you restart your flume agent. But in my
> > case I am switching from one flume agent to another type, let's say from
> > tailsource to AMQP agent. So before I shutdown tailsource I really want
> to
> > make sure WAL is all finished. How do you export the channel size in this
> > case?
> >
> >
> > On 4 December 2012 17:36, Brock Noland <[EMAIL PROTECTED]> wrote:
> >>
> >> Generally you should be fine to bring down the file channel at any
> >> point. That is why we have the WAL.
> >>
> >> If you wanted to be sure the channel was empty, we do export the
> >> channel size (depth).
> >>
> >> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hi,
> >> >
> >> > When it comes to maintenance outage, flume needs to be shutdown and
> >> > offline
> >> > for a while. We want to make sure flume's WAL is completed before we
> >> > shut
> >> > down the process. Is there anyway or any metrics exposed can indicate
> >> > WAL is
> >> > fully processed so that it's safe to bring it down gracefully?
> >> >
> >> > Of coz, the assumption here is that, the source of flume is not
> writing
> >> > any
> >> > new data, flume only has a certain size backlog needs to be cleaned
> up.
> >> >
> >> >
> >> > --
> >> > All the best,
> >> > Shengjie Min
> >> >
> >>
> >>
> >>
> >> --
> >> Apache MRUnit - Unit testing MapReduce -
> >> http://incubator.apache.org/mrunit/
> >
> >
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

--
All the best,
Shengjie Min