Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - flume write ahead log is finished


Copy link to this message
-
Re: flume write ahead log is finished
Brock Noland 2012-12-04, 21:08
You probably want the JSon reporting:
http://flume.apache.org/FlumeUserGuide.html#monitoring

Brock

On Tue, Dec 4, 2012 at 3:02 PM, Shengjie Min <[EMAIL PROTECTED]> wrote:
>>Generally you should be fine to bring down the file channel at any
>>point. That is why we have the WAL.
>
> I understand WAL is there in case you restart your flume agent. But in my
> case I am switching from one flume agent to another type, let's say from
> tailsource to AMQP agent. So before I shutdown tailsource I really want to
> make sure WAL is all finished. How do you export the channel size in this
> case?
>
>
> On 4 December 2012 17:36, Brock Noland <[EMAIL PROTECTED]> wrote:
>>
>> Generally you should be fine to bring down the file channel at any
>> point. That is why we have the WAL.
>>
>> If you wanted to be sure the channel was empty, we do export the
>> channel size (depth).
>>
>> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> >
>> > When it comes to maintenance outage, flume needs to be shutdown and
>> > offline
>> > for a while. We want to make sure flume's WAL is completed before we
>> > shut
>> > down the process. Is there anyway or any metrics exposed can indicate
>> > WAL is
>> > fully processed so that it's safe to bring it down gracefully?
>> >
>> > Of coz, the assumption here is that, the source of flume is not writing
>> > any
>> > new data, flume only has a certain size backlog needs to be cleaned up.
>> >
>> >
>> > --
>> > All the best,
>> > Shengjie Min
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>
>
>
>
> --
> All the best,
> Shengjie Min
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/