Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Consumer in the mirror maker stop fetching logs


Copy link to this message
-
Re: Consumer in the mirror maker stop fetching logs
Actually, do you see log entries like "begin rebalancing" in the consumer
log? Rebalancing can cause sockets to be closed. If you have too many
rebalancing, it's very likely caused by consumer app GC.

Thanks,

Jun
On Sat, May 4, 2013 at 4:58 AM, Helin Xiang <[EMAIL PROTECTED]> wrote:

> Thanks Jun.
>
> I grep the server log, and found  3 kinds of exception on broker.
>
> 1.
> kafka.common.OffsetOutOfRangeException
>
> I don't think this could cause broker closing the socket.
>
>
> 2.
> 579740110 [kafka-processor-6] ERROR kafka.network.Processor  - Closing
> socket for /10.47.7.56 because of error
> java.io.IOException: Broken pipe
>         at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
>         at
> sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:405)
>         at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:506)
>         at kafka.message.FileMessageSet.writeTo(FileMessageSet.scala:102)
>         at kafka.server.MessageSetSend.writeTo(MessageSetSend.scala:53)
>         at kafka.network.MultiSend.writeTo(Transmission.scala:91)
>         at kafka.network.Processor.write(SocketServer.scala:339)
>         at kafka.network.Processor.run(SocketServer.scala:216)
>         at java.lang.Thread.run(Thread.java:662)
>
> 3.
> 580861724 [kafka-processor-13] ERROR kafka.network.Processor  - Closing
> socket for /10.47.7.56 because of error
> java.nio.channels.ClosedChannelException
>         at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
>         at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:486)
>         at kafka.message.FileMessageSet.writeTo(FileMessageSet.scala:102)
>         at kafka.server.MessageSetSend.writeTo(MessageSetSend.scala:53)
>         at kafka.network.MultiSend.writeTo(Transmission.scala:91)
>         at kafka.network.Processor.write(SocketServer.scala:339)
>         at kafka.network.Processor.run(SocketServer.scala:216)
>         at java.lang.Thread.run(Thread.java:662)
>
> probably the second exception may be the cause.
>
> also, I have found the EOFException on consumer side doesn't cause stopping
> fetching every time.
>
>
>
>
> On Fri, May 3, 2013 at 11:50 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > The consumer hit the exception because the broker closed the socket. What
> > does the broker log around the same time say? It should tell you the
> reason
> > why the broker closed the socket.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, May 3, 2013 at 2:34 AM, Helin Xiang <[EMAIL PROTECTED]> wrote:
> >
> > > hi,
> > >
> > > We are doing some mirror-maker test。We currently use version 0.7.2。
> > >
> > > some times,the mirror maker will throw exceptions like this, and then
> > > stopped fetching topics:
> > >
> > > java.io.EOFException: Received -1 when reading from channel, socket has
> > > likely been closed.
> > >      at kafka.utils.Utils$.read(Utils.scala:539)
> > >      at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> > >      at
> kafka.network.Receive$class.readCompletely(Transmission.scala:55)
> > >      at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> > >      at
> > kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
> > >      at
> > > kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:125)
> > >      at
> > kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
> > >      at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> > > [2013-05-02 15:13:05,339] INFO stopping fetcher FetchRunnable-0 to host
> > > kf.a.com (kafka.consumer.FetcherRunnable)
> > >
> > > I used jconsole to double check, and confirmed that the consumer in the
> > > mirror maker stopped fetching logs brokers。I have to stop the mirror
> > maker
> > > and start it over, and it continued fetching logs。
> > >
> > > I start consumers using only 1 consumer thread。
> > >
> > > By the way,there are 16 mirror makers in the consume group。

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB