Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Connection reset by peer


Copy link to this message
-
Re: Connection reset by peer
For Kafka 0.7 in production at Linkedin, we use a heap of size 3G, new gen
256 MB, CMS collector with occupancy of 70%.

Thanks,
Neha

On Sunday, March 24, 2013, Yonghui Zhao wrote:

> Hi Jun,
>
> I used kafka-server-start.sh to start kafka, there is only one jvm setting
> "-Xmx512M“
>
> Do you have some recommend GC setting?   Usually our sever has 32GB or 64GB
> RAM.
>
> 2013/3/22 Jun Rao <[EMAIL PROTECTED]>
>
> > A typical reason for many rebalancing is the consumer side GC. If so, you
> > will see logs in the consume saying sth like "expired session" for ZK.
> > Occasional rebalances are fine. Too many rebalances can slow down the
> > consumption and you will need to tune your GC setting.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Mar 21, 2013 at 11:07 PM, Yonghui Zhao <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Yes, before consumer exception:
> > >
> > > 2013/03/21 12:07:17.909 INFO [ZookeeperConsumerConnector] []
> > > 0_lg-mc-db01.bj-1363784482043-f98c7868 *end rebalancing
> > > consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> > > 2013/03/21 12:07:17.911 INFO [ZookeeperConsumerConnector] []
> > > 0_lg-mc-db01.bj-1363784482043-f98c7868 *begin rebalancing
> > > consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> > > 2013/03/21 12:07:17.934 INFO [FetcherRunnable] [] FetchRunnable-0 start
> > > fetching topic: sms part: 0 offset: 43667888259 from 127.0.0.1:9093
> > > 2013/03/21 12:07:17.940 INFO [SimpleConsumer] [] Reconnect in
> multifetch
> > > due to socket error:
> > > java.nio.channels.*ClosedByInterruptException*
> > >         at java.nio.channels.spi.*AbstractInterruptibleChannel*
> > > .end(AbstractInterruptibleChannel.java:201)
> > >
> > >
> > > 2013/03/21 12:07:17.978 INFO [ZookeeperConsumerConnector] []
> > > 0_lg-mc-db01.bj-1363784482043-f98c7868 *end rebalancing
> > > consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> > > 2013/03/21 12:07:18.004 INFO [FetcherRunnable] [] FetchRunnable-0 start
> > > fetching topic: sms part: 0 offset: 43667888259 from 127.0.0.1:9093
> > > 2013/03/21 12:07:18.066 INFO [ZookeeperConsumerConnector] []
> > > 0_lg-mc-db01.bj-1363784482043-f98c7868 *begin rebalancing consume*r
> > > 0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> > > 2013/03/21 12:07:18.176 INFO [SimpleConsumer] [] Reconnect in
> multifetch
> > > due to socket error:
> > > java.nio.channels.*ClosedByInterruptException*
> > >         at java.nio.channels.spi.*AbstractInterruptibleChannel*
> > > .end(AbstractInterruptibleChannel.java:201)
> > >
> > >
> > > So you think it is normal? How can we avoid this exception?
> > >
> > > I used 4 partitions in kafka,  use only 1 partition?
> > >
> > >
> > >
> > > 2013/3/22 Jun Rao <[EMAIL PROTECTED]>
> > >
> > > > Do you see any rebalances in the consumer? Each rebalance will
> > interrupt
> > > > existing fetcher threads first.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Mar 21, 2013 at 9:40 PM, Yonghui Zhao <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > >
> > > > > The application won't shut down the consumer connector.   The
> > consumer
> > > is
> > > > > always alive.
> > > > >
> > > > > 2013/3/22 Jun Rao <[EMAIL PROTECTED]>
> > > > >
> > > > > > If you use the high level consumer, normally
> > > ClosedByInterruptException
> > > > > > happens because the application calls shutdown on the consumer
> > > > connector.
> > > > > > Is that the case?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, Mar 21, 2013 at 8:38 PM, Yonghui Zhao <
> > [EMAIL PROTECTED]
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > No,  I use java consumer connector, and set 10 seconds timeout.
> > > > > > >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB