Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> 0.8 protocol exception


Copy link to this message
-
Re: 0.8 protocol exception
Thanks Colin, yep you are right I was missing the ShortString way to
represent arrays. All is fine now, I'm able to connect and execute
MetadataRequest/Responses. Moving to ProducerRequest/Responses now :)
On Mon, Jul 8, 2013 at 5:13 PM, Colin Blower <[EMAIL PROTECTED]> wrote:

> I had the exact same problem when I started writing code for the new
> protocol. This is an oddity with the way the protocol spec uses EBNF to
> specify arrays.
>
> Checkout the section on protocol primitives, especially arrays.
>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProtocolPrimitiveTypes
>
> Essentially, each array is preceded by its length as an int32.
>
> With regards to your specific buffer:
> <Buffer 00 00 00 16 00 03 00 00 00 00 00 00 00 03 66 6f 6f 00 07 6d 79
> 54 6f 70 69 63>
>
> You seem to be missing the clientId as well as the array length. For
> comparison the buffer, with the client I wrote, for clientId
> "perl-kafka" and topics "foo" and "myTopic":
>
> 00 00 00 26 00 03 00 00 00 00 00 2a 00 0a 70 65 72 6c 2d 6b 61 66 6b 61
> 00 00 00 02 00 03 66 6f 6f 00 07 6d 79 54 6f 70 69 63
>
>
> On 07/08/2013 04:12 AM, Vinicius Carvalho wrote:
> > Ok, so I've found out the error: The documentation is outdated, the
> > MetadataRequest BNF should be:
> >
> > NumberOfTopics [TopicList]
> >
> > Had to check the scala source code for that.
> >
> > Is there a place with a most to date doc?
> >
> > Regards
> >
> >
> > On Mon, Jul 8, 2013 at 6:42 AM, Vinicius Carvalho <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Hi there. I'm building the 0.8 version of a client to nodejs. I never
> >> coded for node and most of my code is following what the prozees guys
> did
> >> (I'm talking to them on updating the lib)
> >>
> >> But, I'm facing some errors when I test a very simple metadata request.
> >> I'm getting this exception on kafka:
> >>
> >> java.nio.BufferUnderflowException
> >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:145)
> >>  at java.nio.ByteBuffer.get(ByteBuffer.java:694)
> >> at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:38)
> >>  at
> >>
> kafka.api.TopicMetadataRequest$$anonfun$readFrom$1.apply(TopicMetadataRequest.scala:44)
> >> at
> >>
> kafka.api.TopicMetadataRequest$$anonfun$readFrom$1.apply(TopicMetadataRequest.scala:43)
> >>  at
> scala.collection.immutable.Range$ByOne$class.foreach(Range.scala:282)
> >> at scala.collection.immutable.Range$$anon$2.foreach(Range.scala:265)
> >>  at
> >> kafka.api.TopicMetadataRequest$.readFrom(TopicMetadataRequest.scala:43)
> >> at kafka.api.RequestKeys$$anonfun$4.apply(RequestKeys.scala:37)
> >>  at kafka.api.RequestKeys$$anonfun$4.apply(RequestKeys.scala:37)
> >> at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:49)
> >>  at kafka.network.Processor.read(SocketServer.scala:345)
> >> at kafka.network.Processor.run(SocketServer.scala:245)
> >>  at java.lang.Thread.run(Thread.java:722)
> >>
> >> My test sends a metadatarequest (api code 3) using a clientId "foo",
> >> correlationId 0 and topicName "myTopic"
> >>
> >> I don't know where I'm missing something. Please find the js code bellow
> >> and also a java counter version that I've created just for the sake of
> my
> >> lack of experience with node and js. I get the same error with the java
> >> version:
> >>
> >> Request.prototype.toBytes = function() {
> >>     var encoded = new BufferMaker()
> >>         .Int16BE(this.apiKey)
> >>         .Int16BE(API_VERSION)
> >>         .Int32BE(this.correlationId)
> >>         .Int16BE(Buffer.byteLength(this.clientId,'UTF-8'))
> >>         .string(this.clientId)
> >>         .string(this.requestMessage.toBytes()).make();
> >>     var bytes = new
> >> BufferMaker().Int32BE(encoded.length).string(encoded).make();
> >>
> >> MetadataRequest.prototype.toBytes = function () {
> >>     var bytes = new
> >>
> BufferMaker().Int16BE(Buffer.byteLength(this.topicName,'UTF-8')).string(this.topicName).make();
> >>      return bytes;

The intuitive mind is a sacred gift and the
rational mind is a faithful servant. We have
created a society that honors the servant and
has forgotten the gift.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB