Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka 0.8.0 in maven


+
Edward Capriolo 2013-11-05, 03:52
+
Chris Bedford 2013-11-05, 04:08
+
Edward Capriolo 2013-11-05, 04:14
+
Chris Bedford 2013-11-06, 07:53
+
Edward Capriolo 2013-11-07, 03:39
+
Chris Bedford 2013-11-07, 05:16
Copy link to this message
-
Re: Kafka 0.8.0 in maven
One thing I noticed about your code. I thought in kafka 0.8.0 topics are
not created automatically on first message. I do not see anywhere in your
code which creates the topics.

I am creating the topic as part of my tests.

Before I was not setting that property, and getting some of the messages.
Which is really weird. You are using a slightly different kafka then me
(your pom.xml vs mine) as well.

Here is what I have now:

https://github.com/edwardcapriolo/IronCount/blob/iron-ng/src/test/java/com/jointhegrid/ironcount/IronIntegrationTest.java
https://github.com/edwardcapriolo/IronCount/blob/iron-ng/src/test/java/com/jointhegrid/ironcount/IntegrationTest.java

The property I mentioned is making all my tests happy, so that was the
magic bullet for me. Everything else I did in the code above clean up wise
had no effect. I did all the cleanups above in the code and nothing was
working till I switched on that param.
On Thu, Nov 7, 2013 at 12:15 AM, Chris Bedford <[EMAIL PROTECTED]>wrote:

> Do you need to use that configuration to get the tests (as currently
> checked in) to pass ?    I did not find i needed that particular knob
>  (although it is a good one to know about).
>
> and Sorry about your suffering... I can sympathize !
>
>  - cb
>
>
> On Wed, Nov 6, 2013 at 7:38 PM, Edward Capriolo <[EMAIL PROTECTED]
> >wrote:
>
> > After about 5 days of relentless head pounding, and twittling about
> > everything under the sun, I figured it out.
> >
> > If you read to the bottom of this page:
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> >
> > You find:
> >
> > consumerProps.put("auto.offset.reset", "smallest");
> >
> > Now I can bring up the entire stack in jvm and test like I used to. !
> >
> >
> > On Wed, Nov 6, 2013 at 2:52 AM, Chris Bedford <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi, Edward..
> > >
> > > yup .. you are correct.. when we get to a little over 1000 messages the
> > > program was  failing with the  exception stack trace i included below.
> > >
> > > I fixed the test so it passes as long as the consumer gets all messages
> > > sent by the producer.. even if an exception is thrown during shut
> down..
> > >
> > > This isn't as clean as i'd like it to be.  I tried
> > > calling kafkaServer.awaitShutdown();
> > > And I tried inserting some Thread.sleep() calls to give the consumer
> and
> > > producer shut down procedures some time to complete. But I still got
> the
> > > stack trace shown below.
> > >
> > > I don't have time to chase the bug any further.. But I did correct the
> > > test, so you can pull it and see that it passes if you want.
> > >
> > > Maybe we should file a bug on this...?        It might be that I'm
> using
> > > the API incorrectly. I'm not sure at this point.
> > >
> > > anyway, thx for informing me of the issue.
> > >
> > >
> > >
> > >
> > > *Failure due to broken shut down>> >*
> > >
> > > java.io.IOException: Connection reset by peer
> > > at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
> > > at sun.nio.ch.IOUtil.read(IOUtil.java:198)
> > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
> > > at kafka.utils.Utils$.read(Utils.scala:394)
> > > at
> > >
> > >
> >
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> > > at kafka.network.Processor.read(SocketServer.scala:339)
> > > at kafka.network.Processor.run(SocketServer.scala:245)
> > > at java.lang.Thread.run(Thread.java:724)
> > > INFO 2013-11-05 23:45:53,944
> > >
>  ConsumerFetcherThread-RANDOM-GROUP-ID_ubuntu-1383723948022-27b6aba8-0-1
> > > kafka.consumer.SimpleConsumer Reconnect due to socket error:
> > > java.nio.channels.ClosedByInterruptException
> > > at
> > >
> > >
> >
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:402)

 
+
Chris Bedford 2013-11-07, 09:17
+
Edward Capriolo 2013-11-07, 15:03
+
Edward Capriolo 2013-11-05, 04:24
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB