-
Re: flume write ahead log is finishedShengjie Min 2012-12-04, 21:02
>Generally you should be fine to bring down the file channel at any
>point. That is why we have the WAL.
I understand WAL is there in case you restart your flume agent. But in my
case I am switching from one flume agent to another type, let's say from
tailsource to AMQP agent. So before I shutdown tailsource I really want to
make sure WAL is all finished. How do you export the channel size in this
case?
On 4 December 2012 17:36, Brock Noland <[EMAIL PROTECTED]> wrote:
> Generally you should be fine to bring down the file channel at any
> point. That is why we have the WAL.
>
> If you wanted to be sure the channel was empty, we do export the
> channel size (depth).
>
> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > When it comes to maintenance outage, flume needs to be shutdown and
> offline
> > for a while. We want to make sure flume's WAL is completed before we shut
> > down the process. Is there anyway or any metrics exposed can indicate
> WAL is
> > fully processed so that it's safe to bring it down gracefully?
> >
> > Of coz, the assumption here is that, the source of flume is not writing
> any
> > new data, flume only has a certain size backlog needs to be cleaned up.
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
--
All the best,
Shengjie Min
>point. That is why we have the WAL.
I understand WAL is there in case you restart your flume agent. But in my
case I am switching from one flume agent to another type, let's say from
tailsource to AMQP agent. So before I shutdown tailsource I really want to
make sure WAL is all finished. How do you export the channel size in this
case?
On 4 December 2012 17:36, Brock Noland <[EMAIL PROTECTED]> wrote:
> Generally you should be fine to bring down the file channel at any
> point. That is why we have the WAL.
>
> If you wanted to be sure the channel was empty, we do export the
> channel size (depth).
>
> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > When it comes to maintenance outage, flume needs to be shutdown and
> offline
> > for a while. We want to make sure flume's WAL is completed before we shut
> > down the process. Is there anyway or any metrics exposed can indicate
> WAL is
> > fully processed so that it's safe to bring it down gracefully?
> >
> > Of coz, the assumption here is that, the source of flume is not writing
> any
> > new data, flume only has a certain size backlog needs to be cleaned up.
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
--
All the best,
Shengjie Min
