Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Consumer in the mirror maker stop fetching logs


+
Helin Xiang 2013-05-03, 09:35
+
Jun Rao 2013-05-03, 15:50
Copy link to this message
-
Re: Consumer in the mirror maker stop fetching logs
Thanks Jun.

I grep the server log, and found  3 kinds of exception on broker.

1.
kafka.common.OffsetOutOfRangeException

I don't think this could cause broker closing the socket.
2.
579740110 [kafka-processor-6] ERROR kafka.network.Processor  - Closing
socket for /10.47.7.56 because of error
java.io.IOException: Broken pipe
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
        at
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:405)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:506)
        at kafka.message.FileMessageSet.writeTo(FileMessageSet.scala:102)
        at kafka.server.MessageSetSend.writeTo(MessageSetSend.scala:53)
        at kafka.network.MultiSend.writeTo(Transmission.scala:91)
        at kafka.network.Processor.write(SocketServer.scala:339)
        at kafka.network.Processor.run(SocketServer.scala:216)
        at java.lang.Thread.run(Thread.java:662)

3.
580861724 [kafka-processor-13] ERROR kafka.network.Processor  - Closing
socket for /10.47.7.56 because of error
java.nio.channels.ClosedChannelException
        at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:486)
        at kafka.message.FileMessageSet.writeTo(FileMessageSet.scala:102)
        at kafka.server.MessageSetSend.writeTo(MessageSetSend.scala:53)
        at kafka.network.MultiSend.writeTo(Transmission.scala:91)
        at kafka.network.Processor.write(SocketServer.scala:339)
        at kafka.network.Processor.run(SocketServer.scala:216)
        at java.lang.Thread.run(Thread.java:662)

probably the second exception may be the cause.

also, I have found the EOFException on consumer side doesn't cause stopping
fetching every time.
On Fri, May 3, 2013 at 11:50 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> The consumer hit the exception because the broker closed the socket. What
> does the broker log around the same time say? It should tell you the reason
> why the broker closed the socket.
>
> Thanks,
>
> Jun
>
>
> On Fri, May 3, 2013 at 2:34 AM, Helin Xiang <[EMAIL PROTECTED]> wrote:
>
> > hi,
> >
> > We are doing some mirror-maker test。We currently use version 0.7.2。
> >
> > some times,the mirror maker will throw exceptions like this, and then
> > stopped fetching topics:
> >
> > java.io.EOFException: Received -1 when reading from channel, socket has
> > likely been closed.
> >      at kafka.utils.Utils$.read(Utils.scala:539)
> >      at
> >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
> >      at kafka.network.Receive$class.readCompletely(Transmission.scala:55)
> >      at
> >
> >
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> >      at
> kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
> >      at
> > kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:125)
> >      at
> kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
> >      at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
> > [2013-05-02 15:13:05,339] INFO stopping fetcher FetchRunnable-0 to host
> > kf.a.com (kafka.consumer.FetcherRunnable)
> >
> > I used jconsole to double check, and confirmed that the consumer in the
> > mirror maker stopped fetching logs brokers。I have to stop the mirror
> maker
> > and start it over, and it continued fetching logs。
> >
> > I start consumers using only 1 consumer thread。
> >
> > By the way,there are 16 mirror makers in the consume group。
> > I did some calculation and it can be assured that the partition numbers
> are
> > larger than or equal to the consumer threads。
> >
> > Thats all I got。So, I wonder if there was a bug issue for just mirror
> maker
> > or any applications using the high level consumers will behave like this?
> >
> >
> > thanks!
> >
> >
> > --
> > *Best Regards
> >
> > Helin Xiang*
> >
>

--
*Best Regards

向河林*

 
+
Jun Rao 2013-05-06, 01:05