Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Connection reset by peer


Copy link to this message
-
Re: Connection reset by peer
Neha Narkhede 2013-03-26, 15:40
Did you have a gc pause around that time on the server ? What are your
server's current gc settings ?

Thanks,
Neha

On Mon, Mar 25, 2013 at 8:48 PM, Yonghui Zhao <[EMAIL PROTECTED]> wrote:
> Thanks Neha, btw have you seen this exception.  We didn't restart any
> service it happens in deep night.
>
> java.lang.RuntimeException: A broker is already registered on the path
> /brokers/ids/0. This probably indicates that you either have configured a
> brokerid that is already in use, or else you have shutdown this broker and
> restarted it faster than the zookeeper timeout so it appears to be
> re-registering.
>     at
> kafka.server.KafkaZooKeeper.registerBrokerInZk(KafkaZooKeeper.scala:57)
>     at
> kafka.server.KafkaZooKeeper$SessionExpireListener.handleNewSession(KafkaZooKeeper.scala:100)
>     at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472)
>     at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> [2013-03-26 02:07:19,155] INFO re-registering broker info in ZK for broker
> 0 (kafka.server.KafkaZooKeeper)
> [2013-03-26 02:07:19,155] INFO Registering broker /brokers/ids/0
> (kafka.server.KafkaZooKeeper)
> [2013-03-26 02:07:19,611] INFO conflict in /brokers/ids/0 data:
> 127.0.0.1-1364234839275:127.0.0.1:9093 stored data: 127.0.0.1-1364227372971:
> 127.0.0.1:9093 (kafka.utils.ZkUtils$)
> [2013-03-26 02:07:19,611] ERROR Error handling event ZkEvent[New session
> event sent to kafka.server.KafkaZooKeeper$SessionExpireListener@40f8c9bf]
> (org.I0Itec.zkclient.ZkEventThread)
> java.lang.RuntimeException: A broker is already registered on the path
> /brokers/ids/0. This probably indicates that you either have configured a
> brokerid that is already in use, or else you have shutdown this broker and
> restarted it faster than the zookeeper timeout so it appears to be
> re-registering.
>     at
> kafka.server.KafkaZooKeeper.registerBrokerInZk(KafkaZooKeeper.scala:57)
>     at
> kafka.server.KafkaZooKeeper$SessionExpireListener.handleNewSession(KafkaZooKeeper.scala:100)
>     at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472)
>     at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
>
>
>
> 2013/3/26 Neha Narkhede <[EMAIL PROTECTED]>
>
>> That really depends on your consumer application's memory allocation
>> patterns. If it is a thin wrapper over a Kafka consumer, I would imagine
>> you can get away with using CMS for the tenured generation and parallel
>> collector for the new generation with a small heap like 1gb or so.
>>
>> Thanks,
>> Neha
>>
>> On Monday, March 25, 2013, Yonghui Zhao wrote:
>>
>> > Any suggestion on consumer side?
>> > 在 2013-3-25 下午9:49,"Neha Narkhede" <[EMAIL PROTECTED]<javascript:;>
>> > >写道:
>> >
>> > > For Kafka 0.7 in production at Linkedin, we use a heap of size 3G, new
>> > gen
>> > > 256 MB, CMS collector with occupancy of 70%.
>> > >
>> > > Thanks,
>> > > Neha
>> > >
>> > > On Sunday, March 24, 2013, Yonghui Zhao wrote:
>> > >
>> > > > Hi Jun,
>> > > >
>> > > > I used kafka-server-start.sh to start kafka, there is only one jvm
>> > > setting
>> > > > "-Xmx512M“
>> > > >
>> > > > Do you have some recommend GC setting?   Usually our sever has 32GB
>> or
>> > > 64GB
>> > > > RAM.
>> > > >
>> > > > 2013/3/22 Jun Rao <[EMAIL PROTECTED]>
>> > > >
>> > > > > A typical reason for many rebalancing is the consumer side GC. If
>> so,
>> > > you
>> > > > > will see logs in the consume saying sth like "expired session" for
>> > ZK.
>> > > > > Occasional rebalances are fine. Too many rebalances can slow down
>> the
>> > > > > consumption and you will need to tune your GC setting.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Jun
>> > > > >
>> > > > > On Thu, Mar 21, 2013 at 11:07 PM, Yonghui Zhao <
>> > [EMAIL PROTECTED]
>> > > > > >wrote:
>> > > > >
>> > > > > > Yes, before consumer exception:
>> > > > > >
>> > > > > > 2013/03/21 12:07:17.909 INFO [ZookeeperConsumerConnector] []
>> > > > > > 0_lg-mc-db01.bj-1363784482043-f98c7868 *end rebalancing
>> > > > > > consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0