Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - How to make sure flume WAL is finished before shut it down


Copy link to this message
-
Re: How to make sure flume WAL is finished before shut it down
Alexander Alten-Lorenz 2012-12-06, 07:18
moving back to user@

Hi,

You can monitor the directories (usually /tmp/flume*) for not delivered events, or using the reports from the webUI.
https://cwiki.apache.org/confluence/display/FLUME/Monitoring+Flume

Since FlumeOG (0.9x) will not actively developed anymore, we strongly suggest to move into Flume 1.3.x and up. More informations you'll find in our wikie:
https://cwiki.apache.org/confluence/display/FLUME/Home
Regards,
 Alex

On Dec 5, 2012, at 11:07 AM, Shengjie Min <[EMAIL PROTECTED]> wrote:

> I've asked this question in the flume user group, didn't quite get the
> answer.
>
> we are running flume 0.9.4, We want to make sure flume's WAL is completed
> before we shut down the flume process. Is there anyway or any metrics
> exposed can indicate WAL is fully processed so that it's safe to bring it
> down gracefully? How do I know flume has finished all the data?
>
> In our case, we are switching to another input approach, so flume will not
> be getting any inputs, it only has a certain size backlog needs to be
> cleaned up. So it's more like making sure all the data is delivered before
> gracefully shutdown this flume agent forever.
>
> --
> All the best,
> Shengjie Min

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF