Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Connection reset by peer


+
Yonghui Zhao 2013-03-18, 03:25
+
Jun Rao 2013-03-18, 16:25
+
Yonghui Zhao 2013-03-19, 07:29
+
Yonghui Zhao 2013-03-19, 08:35
+
Jun Rao 2013-03-19, 16:15
+
Yonghui Zhao 2013-03-21, 04:26
+
Jun Rao 2013-03-22, 15:06
+
Yonghui Zhao 2013-03-22, 15:46
Copy link to this message
-
Re: Connection reset by peer
Hi Jun,

I used kafka-server-start.sh to start kafka, there is only one jvm setting
"-Xmx512M¡°

Do you have some recommend GC setting?   Usually our sever has 32GB or 64GB
RAM.

2013/3/22 Jun Rao <[EMAIL PROTECTED]>

> A typical reason for many rebalancing is the consumer side GC. If so, you
> will see logs in the consume saying sth like "expired session" for ZK.
> Occasional rebalances are fine. Too many rebalances can slow down the
> consumption and you will need to tune your GC setting.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 21, 2013 at 11:07 PM, Yonghui Zhao <[EMAIL PROTECTED]
> >wrote:
>
> > Yes, before consumer exception:
> >
> > 2013/03/21 12:07:17.909 INFO [ZookeeperConsumerConnector] []
> > 0_lg-mc-db01.bj-1363784482043-f98c7868 *end rebalancing
> > consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> > 2013/03/21 12:07:17.911 INFO [ZookeeperConsumerConnector] []
> > 0_lg-mc-db01.bj-1363784482043-f98c7868 *begin rebalancing
> > consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> > 2013/03/21 12:07:17.934 INFO [FetcherRunnable] [] FetchRunnable-0 start
> > fetching topic: sms part: 0 offset: 43667888259 from 127.0.0.1:9093
> > 2013/03/21 12:07:17.940 INFO [SimpleConsumer] [] Reconnect in multifetch
> > due to socket error:
> > java.nio.channels.*ClosedByInterruptException*
> >         at java.nio.channels.spi.*AbstractInterruptibleChannel*
> > .end(AbstractInterruptibleChannel.java:201)
> >
> >
> > 2013/03/21 12:07:17.978 INFO [ZookeeperConsumerConnector] []
> > 0_lg-mc-db01.bj-1363784482043-f98c7868 *end rebalancing
> > consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> > 2013/03/21 12:07:18.004 INFO [FetcherRunnable] [] FetchRunnable-0 start
> > fetching topic: sms part: 0 offset: 43667888259 from 127.0.0.1:9093
> > 2013/03/21 12:07:18.066 INFO [ZookeeperConsumerConnector] []
> > 0_lg-mc-db01.bj-1363784482043-f98c7868 *begin rebalancing consume*r
> > 0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> > 2013/03/21 12:07:18.176 INFO [SimpleConsumer] [] Reconnect in multifetch
> > due to socket error:
> > java.nio.channels.*ClosedByInterruptException*
> >         at java.nio.channels.spi.*AbstractInterruptibleChannel*
> > .end(AbstractInterruptibleChannel.java:201)
> >
> >
> > So you think it is normal? How can we avoid this exception?
> >
> > I used 4 partitions in kafka,  use only 1 partition£¿
> >
> >
> >
> > 2013/3/22 Jun Rao <[EMAIL PROTECTED]>
> >
> > > Do you see any rebalances in the consumer? Each rebalance will
> interrupt
> > > existing fetcher threads first.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Mar 21, 2013 at 9:40 PM, Yonghui Zhao <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > The application won't shut down the consumer connector.   The
> consumer
> > is
> > > > always alive.
> > > >
> > > > 2013/3/22 Jun Rao <[EMAIL PROTECTED]>
> > > >
> > > > > If you use the high level consumer, normally
> > ClosedByInterruptException
> > > > > happens because the application calls shutdown on the consumer
> > > connector.
> > > > > Is that the case?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, Mar 21, 2013 at 8:38 PM, Yonghui Zhao <
> [EMAIL PROTECTED]
> > >
> > > > > wrote:
> > > > >
> > > > > > No,  I use java consumer connector, and set 10 seconds timeout.
> > > > > >
> > > > > >    ConsumerConfig consumerConfig = new ConsumerConfig(props);
> > > > > >     _consumerConnector =
> > > > > > Consumer.createJavaConsumerConnector(consumerConfig);
> > > > > >    Map<String, Integer> topicCountMap = new HashMap<String,
> > > Integer>();
> > > > > >     topicCountMap.put(_topic, 1);
> > > > > >     Map<String, List<KafkaStream<Message>>> topicMessageStreams =
> > > > > > _consumerConnector
> > > > > >         .createMessageStreams(topicCountMap);
> > > > > >     List<KafkaStream<Message>> streams =
> > > > topicMessageStreams.get(_topic);
> > > > > >     KafkaStream<Message> KafkaStream = streams.iterator().next();
> > > > > >     _consumerIterator = KafkaStream.iterator();

 
+
Neha Narkhede 2013-03-25, 13:49
+
Yonghui Zhao 2013-03-26, 02:14
+
Neha Narkhede 2013-03-26, 03:16
+
Yonghui Zhao 2013-03-26, 03:49
+
Neha Narkhede 2013-03-26, 15:40
+
Yonghui Zhao 2013-03-26, 16:48
+
Neha Narkhede 2013-03-27, 14:52
+
Yonghui Zhao 2013-03-28, 03:35
+
Jun Rao 2013-03-28, 04:54
+
Yonghui Zhao 2013-03-28, 07:23
+
Jun Rao 2013-03-28, 14:52
+
Yonghui Zhao 2013-03-28, 15:21
+
Jun Rao 2013-03-28, 15:25
+
Yonghui Zhao 2013-03-28, 15:32
+
Jun Rao 2013-03-29, 04:03
+
Ranjith Venkatesan 2014-02-02, 06:03