Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Graceful termination of kafka broker after draining all the data consumed


Copy link to this message
-
Re: Graceful termination of kafka broker after draining all the data consumed
Neha Narkhede 2013-01-06, 18:36
In 0.8, we will provide a way for your to shutdown the broker in a
controlled fashion. What that would include is moving all the leaders away
from the broker so that it does not take any more produce requests. Once
that is done, you can shutdown the broker normally. You don't have to wait
until the consumers have pulled the data simply because they can consume
from the new leader.

Thanks,
Neha
On Sun, Jan 6, 2013 at 8:52 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> In 0.7, one way to do this is to use a vip. All producers send data to the
> vip. To decommission a broker,  you first take the broker out of vip so no
> new data will be produced to it. Then you let the consumer drain the data
> (you can use ConsumerOffsetChecker to check if all data has been consumed).
> Finally, you can shut down the broker.
>
> This will be much easier in 0.8 because of replication.
>
> Thanks,
>
> Jun
>
> On Sat, Jan 5, 2013 at 11:34 PM, Bae, Jae Hyeon <[EMAIL PROTECTED]>
> wrote:
>
> > Hi
> >
> > If I want to terminate kafka broker gracefully. Before termination, it
> > should stop receiving the traffic from producers and wait until all
> > data will be consumed.
> >
> > I don't think that kafka 0.7.x is supporting this feature. If I want
> > to implement this feature for myself, could you give me a brief sketch
> > of implementation?
> >
> > Thank you
> > Best, Jae
> >
>