Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Graceful termination of kafka broker after draining all the data consumed


+
Bae, Jae Hyeon 2013-01-06, 07:35
+
Jun Rao 2013-01-06, 16:52
+
Neha Narkhede 2013-01-06, 18:36
Copy link to this message
-
Re: Graceful termination of kafka broker after draining all the data consumed
0.8 sounds really great!

OK, I will try after you release stable build of 0.8

Thank you
Best, Jae

On Sun, Jan 6, 2013 at 10:36 AM, Neha Narkhede <[EMAIL PROTECTED]> wrote:
> In 0.8, we will provide a way for your to shutdown the broker in a
> controlled fashion. What that would include is moving all the leaders away
> from the broker so that it does not take any more produce requests. Once
> that is done, you can shutdown the broker normally. You don't have to wait
> until the consumers have pulled the data simply because they can consume
> from the new leader.
>
> Thanks,
> Neha
>
>
> On Sun, Jan 6, 2013 at 8:52 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> In 0.7, one way to do this is to use a vip. All producers send data to the
>> vip. To decommission a broker,  you first take the broker out of vip so no
>> new data will be produced to it. Then you let the consumer drain the data
>> (you can use ConsumerOffsetChecker to check if all data has been consumed).
>> Finally, you can shut down the broker.
>>
>> This will be much easier in 0.8 because of replication.
>>
>> Thanks,
>>
>> Jun
>>
>> On Sat, Jan 5, 2013 at 11:34 PM, Bae, Jae Hyeon <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Hi
>> >
>> > If I want to terminate kafka broker gracefully. Before termination, it
>> > should stop receiving the traffic from producers and wait until all
>> > data will be consumed.
>> >
>> > I don't think that kafka 0.7.x is supporting this feature. If I want
>> > to implement this feature for myself, could you give me a brief sketch
>> > of implementation?
>> >
>> > Thank you
>> > Best, Jae
>> >
>>

 
+
王国栋 2013-02-17, 10:20
+
Jun Rao 2013-02-18, 16:51
+
王国栋 2013-02-19, 01:41