Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - MirrorMaker doesn't exit on exception?


Copy link to this message
-
Re: MirrorMaker doesn't exit on exception?
Jun Rao 2013-07-03, 05:07
This seems to be a problem. MirrorMaker waits for all threads to finish
before shutting down. In your case, only one thread died. Are you
configuring the producer in sync mode? One potential solution is to run the
producer in async mode. In this mode, the mirror maker thread won't be
killed when there is a socket exception in the producer.

Thanks,

Jun
On Tue, Jul 2, 2013 at 3:51 PM, Homer Strong <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I've noticed that when a broker in the target cluster restarts (i.e.
> immediately comes back online), the MirrorMaker runs into a FATAL exception
> but the process hangs indefinitely instead of exiting. This seems to me
> like undesirable behavior but I'm not sure that it's a bug. I'd like for
> the MirrorMaker process to exit so that it can be restarted, since it is
> run under supervision.
>
> We're on Kafka 0.7.2, and this MirrorMaker uses the sync producer and is
> configured with a broker list.
>
> What I typically find in the logs is something like this:
>
> [2013-07-02 20:56:35,431]  99283617 [mirrormaker-0] INFO
>  kafka.producer.SyncProducer  - Disconnecting from localhost:9094
> [2013-07-02 20:56:35,434]  99283620 [mirrormaker-0] FATAL
> kafka.tools.MirrorMaker$MirrorMakerThread  - %s stream unexpectedly exited.
> java.io.IOException: Broken pipe
> ...
> [2013-07-02 20:56:35,469]  99283655 [mirrormaker-0] INFO
>  kafka.tools.MirrorMaker$MirrorMakerThread  - Stopped thread mirrormaker-0.
>
> and then the process hangs! Only when I kill it does the MirrorMaker
> complete the shutdown process, e.g.:
>
> [2013-07-02 22:34:53,478]  508405 [Thread-2] INFO
>  kafka.consumer.ZookeeperConsumerConnector  -
> MirrorMaker_zkkafka-0.uswest2n.yb0t.com-1372803985231-10bd3698
> ZKConsumerConnector
>  shutting down
> [2013-07-02 22:34:53,479]  508406 [Thread-2] INFO
>  kafka.consumer.ZookeeperTopicEventWatcher  - Shutting down topic event
> watcher.
> [2013-07-02 22:34:53,479]  508406 [ZkClient-EventThread-19-localhost:2181]
> INFO  org.I0Itec.zkclient.ZkEventThread  - Terminate ZkClient event thread.
> [2013-07-02 22:34:53,482]  508409 [main-EventThread] INFO
>  org.apache.zookeeper.ClientCnxn  - EventThread shut down
> [2013-07-02 22:34:53,482]  508409 [Thread-2] INFO
>  org.apache.zookeeper.ZooKeeper  - Session: 0x13ea3d122570085 closed
> [2013-07-02 22:34:53,484]  508411 [Thread-2] INFO
>  kafka.utils.KafkaScheduler  - force shutdown scheduler
> Kafka-consumer-autocommit-
> [2013-07-02 22:34:53,485]  508412 [FetchRunnable-0] INFO
>  kafka.consumer.FetcherRunnable  - FecherRunnable
> Thread[FetchRunnable-0,5,main] interrupted
> [2013-07-02 22:34:53,486]  508413 [FetchRunnable-0] INFO
>  kafka.consumer.FetcherRunnable  - stopping fetcher FetchRunnable-0 to host
> zkkafka-0.uswest2n.yb0t.com
> [2013-07-02 22:34:53,490]  4254397417 [kafka-processor-0] INFO
>  kafka.network.Processor  - Closing socket connection to /10.249.66.10.
> [2013-07-02 22:34:53,499]  508426 [ZkClient-EventThread-12-localhost:2181]
> INFO  org.I0Itec.zkclient.ZkEventThread  - Terminate ZkClient event thread.
> [2013-07-02 22:34:53,503]  508430 [Thread-2] INFO
>  org.apache.zookeeper.ZooKeeper  - Session: 0x13ea3d122570084 closed
> [2013-07-02 22:34:53,503]  508430 [main-EventThread] INFO
>  org.apache.zookeeper.ClientCnxn  - EventThread shut down
> [2013-07-02 22:34:53,504]  508431 [Thread-2] INFO
>  kafka.consumer.ZookeeperConsumerConnector  -
> MirrorMaker_zkkafka-0.uswest2n.yb0t.com-1372803985231-10bd3698
> ZKConsumerConnector
>  shut down completed
> [2013-07-02 22:34:53,505]  508432 [Thread-2] INFO
>  kafka.producer.ProducerPool  - Closing all sync producers
> [2013-07-02 22:34:53,505]  508432 [Thread-2] INFO
>  kafka.producer.SyncProducer  - Disconnecting from localhost:9094
>
>
> Suggestions appreciated. Thanks in advance!
>
> Homer
>