Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Running Kafka in local test mode


Copy link to this message
-
Re: Running Kafka in local test mode

On May 9, 2012, at 2:03pm, Jun Rao wrote:

> Are you starting an embedded broker in your unit test?

That's what I'm trying to do, yes.

> If so, you need to
> call shutdown() on the broker.

I think I do - here's the code from the run() method of my KafakRunnable that I use to run the Kafka broker in my unit test:

        public void run() {
            LOGGER.info("Starting KafkaRunnable...");
            
            KafkaServerStartable kafkaServerStartable = new KafkaServerStartable(_serverConfig);
            kafkaServerStartable.startup();
            _alive = true;
            
            while (!Thread.interrupted()) {
                try {
                    Thread.sleep(100);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }
            
            LOGGER.info("Stopping KafkaRunnable");
            
            kafkaServerStartable.shutdown();
            _alive = false;
            
            LOGGER.info("Exiting KafkaRunnable");
        }

-- Ken
> On Wed, May 9, 2012 at 12:01 PM, Ken Krugler <[EMAIL PROTECTED]>wrote:
>
>> Hi Jun,
>>
>> On May 8, 2012, at 5:40pm, Jun Rao wrote:
>>
>>> The problem is probably that you didn't shut down the broker cleanly (use
>>> kill -15 instead of kill -9).
>>
>> Thanks - though this is for unit tests. So it needs to be something I can
>> do via standard Java code.
>>
>> Is that possible, or does ZK require the Heavy Hammer to get it to
>> terminate?
>>
>> Thanks,
>>
>> -- Ken
>>
>>> On Tue, May 8, 2012 at 5:34 PM, Ken Krugler <[EMAIL PROTECTED]
>>> wrote:
>>>
>>>>
>>>> On May 7, 2012, at 11:10am, Jun Rao wrote:
>>>>
>>>>> Ken,
>>>>>
>>>>> Yes, you need to call ConsumerConnector#shutdown to cleanly shutdown
>> the
>>>>> consumer.
>>>>
>>>> Thanks for the confirmation.
>>>>
>>>>> Clearing ZK data and kafka log should be enough if you want to
>>>>> start from clean. The ZK NoNode exceptions that you saw can happen when
>>>>> some of the ZK paths are created for the very first time. They should
>>>> only
>>>>> show up once though.
>>>>
>>>> But if I delete ZK data at the start of my unit test (to avoid getting
>>>> "broker already registered" errors), then the ZK paths are gone, right?
>>>>
>>>> So these exception would show up every time my test runs, in that case.
>>>>
>>>> Is there a way to avoid the "broker already registered" error and these
>>>> exceptions?
>>>>
>>>> Thanks,
>>>>
>>>> -- Ken
>>>>
>>>>
>>>>> On Sun, May 6, 2012 at 9:53 AM, Ken Krugler <
>> [EMAIL PROTECTED]
>>>>> wrote:
>>>>>
>>>>>> I may have answered my own question…
>>>>>>
>>>>>> Looks like if I call ConsumerConnector#shutdown before interrupting my
>>>>>> consumer Runnable, it works because then
>>>>>> KafkaMessageStream#iterator#hasNext will return false, rather than
>>>> blocking.
>>>>>>
>>>>>> I'm still interested in any examples for the right way to set up/tear
>>>> down
>>>>>> a very temporary Kafka setup for testing.
>>>>>>
>>>>>> For example, I clear out the ZooKeeper data & log dirs before starting
>>>> it
>>>>>> up, in an attempt to avoid occasional errors with "broker already
>>>>>> registered".
>>>>>>
>>>>>> But that in turn seems to trigger Kafka logging about not finding ZK
>>>> nodes
>>>>>> for sessions:
>>>>>>
>>>>>> 12/05/06 09:35:51 INFO server.PrepRequestProcessor: Got user-level
>>>>>> KeeperException when processing sessionid:0x1372301d2120001
>> type:create
>>>>>> cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
>>>>>> Path:/consumers/bixo-storm/ids Error:KeeperErrorCode = NoNode for
>>>>>> /consumers/bixo-storm/ids
>>>>>>
>>>>>> So I assume there's also Kafka state I should be clearing out before
>>>> each
>>>>>> run, right?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> -- Ken
>>>>>>
>>>>>> On May 6, 2012, at 8:21am, Ken Krugler wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I'm trying to run Kakfa in a minimal local test environment, but

http://about.me/kkrugler
+1 530-210-6378
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB