Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Connection reset by peer


Copy link to this message
-
Re: Connection reset by peer
Yonghui Zhao 2013-03-28, 15:32
Will do a check, I just wonder why broker need re-regiester and it failed,
so broker service is stopped.

2013/3/28 Jun Rao <[EMAIL PROTECTED]>

> Do you see lots of ZK session expiration in the broker too? If so, that
> suggests a GC issue in the broker too. So, you may need to tune the GC in
> the broker as well.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 28, 2013 at 8:20 AM, Yonghui Zhao <[EMAIL PROTECTED]>
> wrote:
>
> > Thanks Jun.
> >
> > But I can't understand how consumer GC trigger kafka server issue:
> > java.lang.RuntimeException: A broker is already registered on the path
> > /brokers/ids/0. This probably indicates that you either have configured a
> > brokerid that is already in use, or else you have shutdown this broker
> and
> > restarted it faster than the zookeeper timeout so it appears to be
> > re-registering.
> >
> >
> > 2013/3/28 Jun Rao <[EMAIL PROTECTED]>
> >
> > > The zk session timeout only kicks in if you force kill the consumer.
> > > Otherwise, consumer will close ZK session properly on clean shutdown.
> > >
> > > The problem with GC is that if the consumer pauses for a long time, ZK
> > > server won't receive pings from the client and thus can expire a still
> > > existing session.
> > >
> > > The best thing to do here is to fix the GC issue since it may have
> other
> > > implications. To start with, you probably want to enable GC logging and
> > see
> > > how long and how frequent your GCs are.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Mar 28, 2013 at 12:23 AM, Yonghui Zhao <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > I used zookeeper-3.3.4 in kafka.
> > > >
> > > > Default tickTime is 3 seconds, minSesstionTimeOut is 6 seconds.
> > > > Now I change tickTime to 5 seconds. minSesstionTimeOut to 10 seconds
> > > > But if we change timeout to a larger one,
> > > > "you have shutdown this broker and restarted it faster than the
> > zookeeper
> > > > timeout so it appears to be re-registering."
> > > > this could happened more easily
> > > >
> > > > Do you think consumer GC will affect kafka server and zk connection?
> > > >
> > > >
> > > >
> > > > 2013/3/28 Jun Rao <[EMAIL PROTECTED]>
> > > >
> > > > > Not sure why the re-registration fails. Are you using ZK 3.3.4 or
> > > above?
> > > > >
> > > > > It seems that you consumer still GCs, which is the root cause. So,
> > you
> > > > will
> > > > > need to tune the GC setting further. Another way to avoid ZK
> session
> > > > > timeout is to increase the session timeout config.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Mar 27, 2013 at 8:35 PM, Yonghui Zhao <
> [EMAIL PROTECTED]
> > >
> > > > > wrote:
> > > > >
> > > > > > Now I used GC like this:
> > > > > >
> > > > > > -server -Xms1536m -Xmx1536m -XX:NewSize=128m -XX:MaxNewSize=128m
> > > > > > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> > > > > > -XX:CMSInitiatingOccupancyFraction=70
> > > > > >
> > > > > >
> > > > > > But it still happened.  It seems kafka server reconnect with zk,
> > but
> > > > the
> > > > > > old node was still there. So kafka server stopped.
> > > > > > Can kafka server retry to connect with zk?
> > > > > >
> > > > > >
> > > > > > 2013-03-27 22:15:03,529] INFO Opening socket connection to server
> > > > > > localhost/
> > > > > > 127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn)
> > > > > > [2013-03-27 22:15:03,529] INFO Socket connection established to
> > > > > localhost/
> > > > > > 127.0.0.1:2181, initiating session
> > (org.apache.zookeeper.ClientCnxn)
> > > > > > [2013-03-27 22:15:05,855] INFO Session establishment complete on
> > > server
> > > > > > localhost/127.0.0.1:2181, sessionid = 0x13da6d94abf00aa,
> > negotiated
> > > > > > timeout
> > > > > > = 6000 (org.apache.zookeeper.ClientCnxn)
> > > > > > [2013-03-27 22:15:05,942] INFO zookeeper state changed
> > > (SyncConnected)
> > > > > > (org.I0Itec.zkclient.ZkClient)
> > > > > > [2013-03-27 22:15:14,912] INFO conflict in /brokers/ids/0 data:
> > > > > > 127.0.0.1-1364393691770:127.0.0.1:9093 stored data: null