Hi,

When it comes to maintenance outage, flume needs to be shutdown and offline
for a while. We want to make sure flume's WAL is completed before we shut
down the process. Is there anyway or any metrics exposed can indicate WAL
is fully processed so that it's safe to bring it down gracefully?

Of coz, the assumption here is that, the source of flume is not writing any
new data, flume only has a certain size backlog needs to be cleaned up.

--
All the best,
Shengjie Min

Generally you should be fine to bring down the file channel at any
point. That is why we have the WAL.

If you wanted to be sure the channel was empty, we do export the
channel size (depth).

On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <[EMAIL PROTECTED]> wrote:
> Hi,
>
> When it comes to maintenance outage, flume needs to be shutdown and offline
> for a while. We want to make sure flume's WAL is completed before we shut
> down the process. Is there anyway or any metrics exposed can indicate WAL is
> fully processed so that it's safe to bring it down gracefully?
>
> Of coz, the assumption here is that, the source of flume is not writing any
> new data, flume only has a certain size backlog needs to be cleaned up.
>
>
> --
> All the best,
> Shengjie Min
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

>Generally you should be fine to bring down the file channel at any
>point. That is why we have the WAL.

I understand WAL is there in case you restart your flume agent. But in my
case I am switching from one flume agent to another type, let's say from
tailsource to AMQP agent. So before I shutdown tailsource I really want to
make sure WAL is all finished. How do you export the channel size in this
case?
On 4 December 2012 17:36, Brock Noland <[EMAIL PROTECTED]> wrote:

> Generally you should be fine to bring down the file channel at any
> point. That is why we have the WAL.
>
> If you wanted to be sure the channel was empty, we do export the
> channel size (depth).
>
> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > When it comes to maintenance outage, flume needs to be shutdown and
> offline
> > for a while. We want to make sure flume's WAL is completed before we shut
> > down the process. Is there anyway or any metrics exposed can indicate
> WAL is
> > fully processed so that it's safe to bring it down gracefully?
> >
> > Of coz, the assumption here is that, the source of flume is not writing
> any
> > new data, flume only has a certain size backlog needs to be cleaned up.
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

--
All the best,
Shengjie Min

You probably want the JSon reporting:
http://flume.apache.org/FlumeUserGuide.html#monitoring

Brock

On Tue, Dec 4, 2012 at 3:02 PM, Shengjie Min <[EMAIL PROTECTED]> wrote:
>>Generally you should be fine to bring down the file channel at any
>>point. That is why we have the WAL.
>
> I understand WAL is there in case you restart your flume agent. But in my
> case I am switching from one flume agent to another type, let's say from
> tailsource to AMQP agent. So before I shutdown tailsource I really want to
> make sure WAL is all finished. How do you export the channel size in this
> case?
>
>
> On 4 December 2012 17:36, Brock Noland <[EMAIL PROTECTED]> wrote:
>>
>> Generally you should be fine to bring down the file channel at any
>> point. That is why we have the WAL.
>>
>> If you wanted to be sure the channel was empty, we do export the
>> channel size (depth).
>>
>> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> >
>> > When it comes to maintenance outage, flume needs to be shutdown and
>> > offline
>> > for a while. We want to make sure flume's WAL is completed before we
>> > shut
>> > down the process. Is there anyway or any metrics exposed can indicate
>> > WAL is
>> > fully processed so that it's safe to bring it down gracefully?
>> >
>> > Of coz, the assumption here is that, the source of flume is not writing
>> > any
>> > new data, flume only has a certain size backlog needs to be cleaned up.
>> >
>> >
>> > --
>> > All the best,
>> > Shengjie Min
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>
>
>
>
> --
> All the best,
> Shengjie Min
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

sorry, I should've mentioned this at the beginning, we are running
flume 0.9.4-cdh3u4. It's not 1.3 unfortunately.

Shengjie
On 4 December 2012 21:08, Brock Noland <[EMAIL PROTECTED]> wrote:

> You probably want the JSon reporting:
> http://flume.apache.org/FlumeUserGuide.html#monitoring
>
> Brock
>
> On Tue, Dec 4, 2012 at 3:02 PM, Shengjie Min <[EMAIL PROTECTED]> wrote:
> >>Generally you should be fine to bring down the file channel at any
> >>point. That is why we have the WAL.
> >
> > I understand WAL is there in case you restart your flume agent. But in my
> > case I am switching from one flume agent to another type, let's say from
> > tailsource to AMQP agent. So before I shutdown tailsource I really want
> to
> > make sure WAL is all finished. How do you export the channel size in this
> > case?
> >
> >
> > On 4 December 2012 17:36, Brock Noland <[EMAIL PROTECTED]> wrote:
> >>
> >> Generally you should be fine to bring down the file channel at any
> >> point. That is why we have the WAL.
> >>
> >> If you wanted to be sure the channel was empty, we do export the
> >> channel size (depth).
> >>
> >> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hi,
> >> >
> >> > When it comes to maintenance outage, flume needs to be shutdown and
> >> > offline
> >> > for a while. We want to make sure flume's WAL is completed before we
> >> > shut
> >> > down the process. Is there anyway or any metrics exposed can indicate
> >> > WAL is
> >> > fully processed so that it's safe to bring it down gracefully?
> >> >
> >> > Of coz, the assumption here is that, the source of flume is not
> writing
> >> > any
> >> > new data, flume only has a certain size backlog needs to be cleaned
> up.
> >> >
> >> >
> >> > --
> >> > All the best,
> >> > Shengjie Min
> >> >
> >>
> >>
> >>
> >> --
> >> Apache MRUnit - Unit testing MapReduce -
> >> http://incubator.apache.org/mrunit/
> >
> >
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>

--
All the best,
Shengjie Min