Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Connection reset by peer


+
Yonghui Zhao 2013-03-18, 03:25
+
Jun Rao 2013-03-18, 16:25
+
Yonghui Zhao 2013-03-19, 07:29
+
Yonghui Zhao 2013-03-19, 08:35
+
Jun Rao 2013-03-19, 16:15
+
Yonghui Zhao 2013-03-21, 04:26
Copy link to this message
-
Re: Connection reset by peer
A typical reason for many rebalancing is the consumer side GC. If so, you
will see logs in the consume saying sth like "expired session" for ZK.
Occasional rebalances are fine. Too many rebalances can slow down the
consumption and you will need to tune your GC setting.

Thanks,

Jun

On Thu, Mar 21, 2013 at 11:07 PM, Yonghui Zhao <[EMAIL PROTECTED]>wrote:

> Yes, before consumer exception:
>
> 2013/03/21 12:07:17.909 INFO [ZookeeperConsumerConnector] []
> 0_lg-mc-db01.bj-1363784482043-f98c7868 *end rebalancing
> consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> 2013/03/21 12:07:17.911 INFO [ZookeeperConsumerConnector] []
> 0_lg-mc-db01.bj-1363784482043-f98c7868 *begin rebalancing
> consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> 2013/03/21 12:07:17.934 INFO [FetcherRunnable] [] FetchRunnable-0 start
> fetching topic: sms part: 0 offset: 43667888259 from 127.0.0.1:9093
> 2013/03/21 12:07:17.940 INFO [SimpleConsumer] [] Reconnect in multifetch
> due to socket error:
> java.nio.channels.*ClosedByInterruptException*
>         at java.nio.channels.spi.*AbstractInterruptibleChannel*
> .end(AbstractInterruptibleChannel.java:201)
>
>
> 2013/03/21 12:07:17.978 INFO [ZookeeperConsumerConnector] []
> 0_lg-mc-db01.bj-1363784482043-f98c7868 *end rebalancing
> consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> 2013/03/21 12:07:18.004 INFO [FetcherRunnable] [] FetchRunnable-0 start
> fetching topic: sms part: 0 offset: 43667888259 from 127.0.0.1:9093
> 2013/03/21 12:07:18.066 INFO [ZookeeperConsumerConnector] []
> 0_lg-mc-db01.bj-1363784482043-f98c7868 *begin rebalancing consume*r
> 0_lg-mc-db01.bj-1363784482043-f98c7868 try #0
> 2013/03/21 12:07:18.176 INFO [SimpleConsumer] [] Reconnect in multifetch
> due to socket error:
> java.nio.channels.*ClosedByInterruptException*
>         at java.nio.channels.spi.*AbstractInterruptibleChannel*
> .end(AbstractInterruptibleChannel.java:201)
>
>
> So you think it is normal? How can we avoid this exception?
>
> I used 4 partitions in kafka,  use only 1 partition?
>
>
>
> 2013/3/22 Jun Rao <[EMAIL PROTECTED]>
>
> > Do you see any rebalances in the consumer? Each rebalance will interrupt
> > existing fetcher threads first.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Mar 21, 2013 at 9:40 PM, Yonghui Zhao <[EMAIL PROTECTED]>
> > wrote:
> >
> > > The application won't shut down the consumer connector.   The consumer
> is
> > > always alive.
> > >
> > > 2013/3/22 Jun Rao <[EMAIL PROTECTED]>
> > >
> > > > If you use the high level consumer, normally
> ClosedByInterruptException
> > > > happens because the application calls shutdown on the consumer
> > connector.
> > > > Is that the case?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Mar 21, 2013 at 8:38 PM, Yonghui Zhao <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > >
> > > > > No,  I use java consumer connector, and set 10 seconds timeout.
> > > > >
> > > > >    ConsumerConfig consumerConfig = new ConsumerConfig(props);
> > > > >     _consumerConnector =
> > > > > Consumer.createJavaConsumerConnector(consumerConfig);
> > > > >    Map<String, Integer> topicCountMap = new HashMap<String,
> > Integer>();
> > > > >     topicCountMap.put(_topic, 1);
> > > > >     Map<String, List<KafkaStream<Message>>> topicMessageStreams =
> > > > > _consumerConnector
> > > > >         .createMessageStreams(topicCountMap);
> > > > >     List<KafkaStream<Message>> streams =
> > > topicMessageStreams.get(_topic);
> > > > >     KafkaStream<Message> KafkaStream = streams.iterator().next();
> > > > >     _consumerIterator = KafkaStream.iterator();
> > > > >
> > > > > 2013/3/21 Jun Rao <[EMAIL PROTECTED]>
> > > > >
> > > > > > So, it seems that your consume thread was interrupted and
> therefore
> > > the
> > > > > > socket channel was closed. Are you using SimpleConsumer?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Mar 20, 2013 at 9:25 PM, Yonghui Zhao <
> > [EMAIL PROTECTED]
> > > >

 
+
Yonghui Zhao 2013-03-22, 15:46
+
Yonghui Zhao 2013-03-25, 06:14
+
Neha Narkhede 2013-03-25, 13:49
+
Yonghui Zhao 2013-03-26, 02:14
+
Neha Narkhede 2013-03-26, 03:16
+
Yonghui Zhao 2013-03-26, 03:49
+
Neha Narkhede 2013-03-26, 15:40
+
Yonghui Zhao 2013-03-26, 16:48
+
Neha Narkhede 2013-03-27, 14:52
+
Yonghui Zhao 2013-03-28, 03:35
+
Jun Rao 2013-03-28, 04:54
+
Yonghui Zhao 2013-03-28, 07:23
+
Jun Rao 2013-03-28, 14:52
+
Yonghui Zhao 2013-03-28, 15:21
+
Jun Rao 2013-03-28, 15:25
+
Yonghui Zhao 2013-03-28, 15:32
+
Jun Rao 2013-03-29, 04:03
+
Ranjith Venkatesan 2014-02-02, 06:03