Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Running Kafka in local test mode


Copy link to this message
-
Re: Running Kafka in local test mode
I may have answered my own question…

Looks like if I call ConsumerConnector#shutdown before interrupting my consumer Runnable, it works because then KafkaMessageStream#iterator#hasNext will return false, rather than blocking.

I'm still interested in any examples for the right way to set up/tear down a very temporary Kafka setup for testing.

For example, I clear out the ZooKeeper data & log dirs before starting it up, in an attempt to avoid occasional errors with "broker already registered".

But that in turn seems to trigger Kafka logging about not finding ZK nodes for sessions:

12/05/06 09:35:51 INFO server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1372301d2120001 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/consumers/bixo-storm/ids Error:KeeperErrorCode = NoNode for /consumers/bixo-storm/ids

So I assume there's also Kafka state I should be clearing out before each run, right?

Thanks,

-- Ken

On May 6, 2012, at 8:21am, Ken Krugler wrote:

> Hi all,
>
> I'm trying to run Kakfa in a minimal local test environment, but having issues gracefully shutting down.
>
> I can start up ZooKeeper/Kafka, and it's running fine.
>
> But when I try to shut it all down, I'm having trouble cleanly terminating the consumers.
>
> I think the issue is that they're blocking on ConsumerIterator.makeNext(), which doesn't seem to be paying attention to being interrupted.
>
> So then I proceed with cleaning up everything else, and shutting down the Kafka broker.
>
> Which in turn triggers a kafka.consumer.ConsumerTimeoutException from the pending hasNext() call in my consumer Runnable.
>
> What's the clean way to set up/tear down a ZooKeeper/Kafka setup that's being used indirectly by the test of another project?
>
> Thanks!
>
> -- Ken
>
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Mahout & Solr
>
>
>
>

--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr

--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB