Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Connection reset by peer


Copy link to this message
-
Re: Connection reset by peer
Thanks Jun.

But I can't understand how consumer GC trigger kafka server issue:
java.lang.RuntimeException: A broker is already registered on the path
/brokers/ids/0. This probably indicates that you either have configured a
brokerid that is already in use, or else you have shutdown this broker and
restarted it faster than the zookeeper timeout so it appears to be
re-registering.
2013/3/28 Jun Rao <[EMAIL PROTECTED]>

> The zk session timeout only kicks in if you force kill the consumer.
> Otherwise, consumer will close ZK session properly on clean shutdown.
>
> The problem with GC is that if the consumer pauses for a long time, ZK
> server won't receive pings from the client and thus can expire a still
> existing session.
>
> The best thing to do here is to fix the GC issue since it may have other
> implications. To start with, you probably want to enable GC logging and see
> how long and how frequent your GCs are.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 28, 2013 at 12:23 AM, Yonghui Zhao <[EMAIL PROTECTED]
> >wrote:
>
> > I used zookeeper-3.3.4 in kafka.
> >
> > Default tickTime is 3 seconds, minSesstionTimeOut is 6 seconds.
> > Now I change tickTime to 5 seconds. minSesstionTimeOut to 10 seconds
> > But if we change timeout to a larger one,
> > "you have shutdown this broker and restarted it faster than the zookeeper
> > timeout so it appears to be re-registering."
> > this could happened more easily
> >
> > Do you think consumer GC will affect kafka server and zk connection?
> >
> >
> >
> > 2013/3/28 Jun Rao <[EMAIL PROTECTED]>
> >
> > > Not sure why the re-registration fails. Are you using ZK 3.3.4 or
> above?
> > >
> > > It seems that you consumer still GCs, which is the root cause. So, you
> > will
> > > need to tune the GC setting further. Another way to avoid ZK session
> > > timeout is to increase the session timeout config.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Mar 27, 2013 at 8:35 PM, Yonghui Zhao <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Now I used GC like this:
> > > >
> > > > -server -Xms1536m -Xmx1536m -XX:NewSize=128m -XX:MaxNewSize=128m
> > > > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> > > > -XX:CMSInitiatingOccupancyFraction=70
> > > >
> > > >
> > > > But it still happened.  It seems kafka server reconnect with zk, but
> > the
> > > > old node was still there. So kafka server stopped.
> > > > Can kafka server retry to connect with zk?
> > > >
> > > >
> > > > 2013-03-27 22:15:03,529] INFO Opening socket connection to server
> > > > localhost/
> > > > 127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn)
> > > > [2013-03-27 22:15:03,529] INFO Socket connection established to
> > > localhost/
> > > > 127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
> > > > [2013-03-27 22:15:05,855] INFO Session establishment complete on
> server
> > > > localhost/127.0.0.1:2181, sessionid = 0x13da6d94abf00aa, negotiated
> > > > timeout
> > > > = 6000 (org.apache.zookeeper.ClientCnxn)
> > > > [2013-03-27 22:15:05,942] INFO zookeeper state changed
> (SyncConnected)
> > > > (org.I0Itec.zkclient.ZkClient)
> > > > [2013-03-27 22:15:14,912] INFO conflict in /brokers/ids/0 data:
> > > > 127.0.0.1-1364393691770:127.0.0.1:9093 stored data: null
> > > > (kafka.utils.ZkUtils$)
> > > > [2013-03-27 22:15:14,942] ERROR Error handling event ZkEvent[New
> > session
> > > > event sent to
> > kafka.server.KafkaZooKeeper$SessionExpireListener@18f389bc
> > > ]
> > > > (org.I0Itec.zkclient.ZkEventThread)
> > > > java.lang.RuntimeException: A broker is already registered on the
> path
> > > > /brokers/ids/0. This probably indicates that you either have
> > configured a
> > > > brokerid that is already in use, or else you have shutdown this
> broker
> > > and
> > > > restarted it faster than the zookeeper timeout so it appears to be
> > > > re-registering.
> > > >     at
> > > >
> kafka.server.KafkaZooKeeper.registerBrokerInZk(KafkaZooKeeper.scala:57)
> > > >     at
> > > >
> > > >
> > >
> >
> kafka.server.KafkaZooKeeper$SessionExpireListener.handleNewSession(KafkaZooKeeper.scala:100)