I defined a customized partitioner:

class ExpertSearchCollectionPartitioner implements Partitioner<Long> {

  @Override

  public int partition(Long key, int numPartitions) {

    System.out.print("    partition: " + key + ", " + numPartitions +
", "+ (key.hashCode() % numPartitions) +
"\n");

    return (int) (key % (long) numPartitions);

  }

};
and in the producer:

props.put("partitioner.class",  ExpertSearchCollectionPartitioner.class
.getName());

producer = new Producer<Long, String>(new ProducerConfig(props));

producer.send(new KeyedMessage<Long, String>(

              KAFKA_EXPERTSEARCHCOLLECTION_TOPIC,

              new Long(1),

              new String("I love kafka" + count)));
however, when I run the producer, I always got the following error:
Exception in thread "main" java.lang.NoSuchMethodException:
com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionPartitioner.<init>(kafka.utils.VerifiableProperties)

at java.lang.Class.getConstructor0(Class.java:2721)

at java.lang.Class.getConstructor(Class.java:1674)

at kafka.utils.Utils$.createObject(Utils.scala:457)

at kafka.producer.Producer.<init>(Producer.scala:61)

at kafka.javaapi.producer.Producer.<init>(Producer.scala:25)

at
com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionKafkaProducer.<init>(ExpertSearchCollectionKafkaProducer.java:70)

at
com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionKafkaProducer.main(ExpertSearchCollectionKafkaProducer.java:91)

May I ask what I am missing? Is there anywhere an example java code to
customize the partitioner?

Thanks alot!
JAne

 

Hi Jane,

This error is
java.lang.NoSuchMethodException:
com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionPartitio
ner.<init>(kafka.utils.VerifiableProperties)

It means that you need to have a constructor which takes a single argument,
VerifiableProperties. This is a way to pass any configuration from the
producer properties to your partitioner instance.

Sorry there is no real documentation for this as it is new to 0.8 and we
haven't finished the 0.8 documentation.

-Jay

On Thu, Jan 17, 2013 at 5:29 PM, Jane Wang <[EMAIL PROTECTED]> wrote:

> java.lang.NoSuchMethodException:
>
> com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionPartitioner.<init>(kafka.utils.VerifiableProperties)
>

 

I also tried with the constructor
ExpertSearchCollectionPartitioner(VerifiableProperties props) {
}

Still the same error.

I also tried the constructor without any parameter and with parameter Long.
None of them worked.

On Thu, Jan 17, 2013 at 7:08 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Hi Jane,
>
> This error is
> java.lang.NoSuchMethodException:
> com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionPartitio
> ner.<init>(kafka.utils.VerifiableProperties)
>
> It means that you need to have a constructor which takes a single argument,
> VerifiableProperties. This is a way to pass any configuration from the
> producer properties to your partitioner instance.
>
> Sorry there is no real documentation for this as it is new to 0.8 and we
> haven't finished the 0.8 documentation.
>
> -Jay
>
> On Thu, Jan 17, 2013 at 5:29 PM, Jane Wang <[EMAIL PROTECTED]> wrote:
>
> > java.lang.NoSuchMethodException:
> >
> >
> com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionPartitioner.<init>(kafka.utils.VerifiableProperties)
> >
>

 

finally, I made the partitioner working after changing the partition key to
be string.
and add the key.serializer.class to the config props. thanks for the help.
On Thu, Jan 17, 2013 at 7:57 PM, Jane Wang <[EMAIL PROTECTED]> wrote:

> I also tried with the constructor
> ExpertSearchCollectionPartitioner(VerifiableProperties props) {
> }
>
> Still the same error.
>
> I also tried the constructor without any parameter and with parameter Long.
> None of them worked.
>
>
>
>
>
> On Thu, Jan 17, 2013 at 7:08 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
>> Hi Jane,
>>
>> This error is
>> java.lang.NoSuchMethodException:
>> com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionPartitio
>> ner.<init>(kafka.utils.VerifiableProperties)
>>
>> It means that you need to have a constructor which takes a single
>> argument,
>> VerifiableProperties. This is a way to pass any configuration from the
>> producer properties to your partitioner instance.
>>
>> Sorry there is no real documentation for this as it is new to 0.8 and we
>> haven't finished the 0.8 documentation.
>>
>> -Jay
>>
>> On Thu, Jan 17, 2013 at 5:29 PM, Jane Wang <[EMAIL PROTECTED]> wrote:
>>
>> > java.lang.NoSuchMethodException:
>> >
>> >
>> com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionPartitioner.<init>(kafka.utils.VerifiableProperties)
>> >
>>
>
>

 

Hmmm, so I would like to turn this into a bug for us, since at the very
least the error is confusing, but I am not quite sure what was going on. At
least one issue is the error doesn't make it clear whether this is for the
key serializer or value serializer and could be cleaned up. Let me try to
reproduce...

-Jay
On Fri, Jan 18, 2013 at 9:26 AM, Jane Wang <[EMAIL PROTECTED]> wrote:

> finally, I made the partitioner working after changing the partition key to
> be string.
> and add the key.serializer.class to the config props. thanks for the help.
>
>
> On Thu, Jan 17, 2013 at 7:57 PM, Jane Wang <[EMAIL PROTECTED]> wrote:
>
> > I also tried with the constructor
> > ExpertSearchCollectionPartitioner(VerifiableProperties props) {
> > }
> >
> > Still the same error.
> >
> > I also tried the constructor without any parameter and with parameter
> Long.
> > None of them worked.
> >
> >
> >
> >
> >
> > On Thu, Jan 17, 2013 at 7:08 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Jane,
> >>
> >> This error is
> >> java.lang.NoSuchMethodException:
> >> com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionPartitio
> >> ner.<init>(kafka.utils.VerifiableProperties)
> >>
> >> It means that you need to have a constructor which takes a single
> >> argument,
> >> VerifiableProperties. This is a way to pass any configuration from the
> >> producer properties to your partitioner instance.
> >>
> >> Sorry there is no real documentation for this as it is new to 0.8 and we
> >> haven't finished the 0.8 documentation.
> >>
> >> -Jay
> >>
> >> On Thu, Jan 17, 2013 at 5:29 PM, Jane Wang <[EMAIL PROTECTED]> wrote:
> >>
> >> > java.lang.NoSuchMethodException:
> >> >
> >> >
> >>
> com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionPartitioner.<init>(kafka.utils.VerifiableProperties)
> >> >
> >>
> >
> >
>