Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Connection reset by peer


Copy link to this message
-
Re: Connection reset by peer
The kafka-server-start.sh script doesn't have the mentioned GC
settings and heap size configured. However, probably doing that is a
good idea.

Thanks,
Neha

On Tue, Mar 26, 2013 at 9:47 AM, Yonghui Zhao <[EMAIL PROTECTED]> wrote:
> kafka server is started by bin/kafka-server-start.sh.  No gc setting.
> 在 2013-3-26 下午11:40,"Neha Narkhede" <[EMAIL PROTECTED]>写道:
>
>> Did you have a gc pause around that time on the server ? What are your
>> server's current gc settings ?
>>
>> Thanks,
>> Neha
>>
>> On Mon, Mar 25, 2013 at 8:48 PM, Yonghui Zhao <[EMAIL PROTECTED]>
>> wrote:
>> > Thanks Neha, btw have you seen this exception.  We didn't restart any
>> > service it happens in deep night.
>> >
>> > java.lang.RuntimeException: A broker is already registered on the path
>> > /brokers/ids/0. This probably indicates that you either have configured a
>> > brokerid that is already in use, or else you have shutdown this broker
>> and
>> > restarted it faster than the zookeeper timeout so it appears to be
>> > re-registering.
>> >     at
>> > kafka.server.KafkaZooKeeper.registerBrokerInZk(KafkaZooKeeper.scala:57)
>> >     at
>> >
>> kafka.server.KafkaZooKeeper$SessionExpireListener.handleNewSession(KafkaZooKeeper.scala:100)
>> >     at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472)
>> >     at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
>> > [2013-03-26 02:07:19,155] INFO re-registering broker info in ZK for
>> broker
>> > 0 (kafka.server.KafkaZooKeeper)
>> > [2013-03-26 02:07:19,155] INFO Registering broker /brokers/ids/0
>> > (kafka.server.KafkaZooKeeper)
>> > [2013-03-26 02:07:19,611] INFO conflict in /brokers/ids/0 data:
>> > 127.0.0.1-1364234839275:127.0.0.1:9093 stored data:
>> 127.0.0.1-1364227372971:
>> > 127.0.0.1:9093 (kafka.utils.ZkUtils$)
>> > [2013-03-26 02:07:19,611] ERROR Error handling event ZkEvent[New session
>> > event sent to kafka.server.KafkaZooKeeper$SessionExpireListener@40f8c9bf
>> ]
>> > (org.I0Itec.zkclient.ZkEventThread)
>> > java.lang.RuntimeException: A broker is already registered on the path
>> > /brokers/ids/0. This probably indicates that you either have configured a
>> > brokerid that is already in use, or else you have shutdown this broker
>> and
>> > restarted it faster than the zookeeper timeout so it appears to be
>> > re-registering.
>> >     at
>> > kafka.server.KafkaZooKeeper.registerBrokerInZk(KafkaZooKeeper.scala:57)
>> >     at
>> >
>> kafka.server.KafkaZooKeeper$SessionExpireListener.handleNewSession(KafkaZooKeeper.scala:100)
>> >     at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472)
>> >     at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
>> >
>> >
>> >
>> > 2013/3/26 Neha Narkhede <[EMAIL PROTECTED]>
>> >
>> >> That really depends on your consumer application's memory allocation
>> >> patterns. If it is a thin wrapper over a Kafka consumer, I would imagine
>> >> you can get away with using CMS for the tenured generation and parallel
>> >> collector for the new generation with a small heap like 1gb or so.
>> >>
>> >> Thanks,
>> >> Neha
>> >>
>> >> On Monday, March 25, 2013, Yonghui Zhao wrote:
>> >>
>> >> > Any suggestion on consumer side?
>> >> > 在 2013-3-25 下午9:49,"Neha Narkhede" <[EMAIL PROTECTED]
>> <javascript:;>
>> >> > >写道:
>> >> >
>> >> > > For Kafka 0.7 in production at Linkedin, we use a heap of size 3G,
>> new
>> >> > gen
>> >> > > 256 MB, CMS collector with occupancy of 70%.
>> >> > >
>> >> > > Thanks,
>> >> > > Neha
>> >> > >
>> >> > > On Sunday, March 24, 2013, Yonghui Zhao wrote:
>> >> > >
>> >> > > > Hi Jun,
>> >> > > >
>> >> > > > I used kafka-server-start.sh to start kafka, there is only one jvm
>> >> > > setting
>> >> > > > "-Xmx512M“
>> >> > > >
>> >> > > > Do you have some recommend GC setting?   Usually our sever has
>> 32GB
>> >> or
>> >> > > 64GB
>> >> > > > RAM.
>> >> > > >
>> >> > > > 2013/3/22 Jun Rao <[EMAIL PROTECTED]>
>> >> > > >
>> >> > > > > A typical reason for many rebalancing is the consumer side GC.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB