Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> consumer reaction to auto-create topics


+
David Morales de Frías 2014-03-06, 10:11
+
Jun Rao 2014-03-06, 17:28
+
David Morales de Frías 2014-03-06, 17:35
+
Jun Rao 2014-03-06, 17:53
+
Neha Narkhede 2014-03-06, 17:56
+
David Morales de Frías 2014-03-07, 08:23
+
David Morales de Frías 2014-03-13, 09:43
+
Guozhang Wang 2014-03-13, 16:04
+
David Morales de Frías 2014-03-13, 16:12
Copy link to this message
-
Re: consumer reaction to auto-create topics (0.8.1)
One system I am working on we have something similar however we are
creating topics like how the TopicCommand does but different.  We keep
those operations persisted in a meta store so consumers that have to-do
something know that everything is all ready for them (as often it is not
just Kafka) for them to-do "their thing".  In our case we are spinning up
consumers within our Mesos slaves... we don't want that to happen unless
there is a topic present and is showing up in our monitoring first (test
messages going in first).

Think of it like the producer having to request permission for a new topic
to be created and blocking waiting for that to happen (first one creates
the topic and the other request ack back right away as topic is created).
 There may be good reason for that topic to not be created i.e. half your
hardware just "conked" out and Mesos is rebalancing all non priority
services killing them to start up more priority services... you also have
to make sure then that no new low priority services are created.

I think a lot of this comes down to different domain specific build systems
and implementing these behaviors within them, once integration hooks are
understood (and easily available). Maybe we need to make the command tools
more plug-able?  Without trying to become "plug-able happy" we could
broaden the convo looking at command tools as a whole.  There is thread
going for that lets lump this ticket into it? At least so it can solved as
part of a larger piece?  Or maybe just make it more like an API and the
project ships with a shell wrapper for the API?  Not far off from that now.

There was also a patch for a different restart mechanism
https://issues.apache.org/jira/browse/KAFKA-1300  that looked interesting
that would be part of it too I would think? So folks could share parts of
the implementation but have a way for it to work smoothly in their build
process? Tools out of the box and then also ones that can be plugged in.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Thu, Mar 13, 2014 at 12:12 PM, David Morales de Frías <
[EMAIL PROTECTED]> wrote:
 
+
Guozhang Wang 2014-03-13, 17:21
+
Neha Narkhede 2014-03-13, 20:29
+
Jun Rao 2014-03-14, 04:00