Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: How to get partition information from Kafka Stream?


Copy link to this message
-
Re: How to get partition information from Kafka Stream?
Philip O'Toole 2013-06-27, 15:34
Could you just use a topic for each stage? The nice thing about topics
is you don't even have to tell the Kafka cluster about the topics
beforehand, so if you a new stage appears, just start writing to that
topic.

Kafka is great. :-)

Phiilp

On Thu, Jun 27, 2013 at 7:27 AM, Hussain Pirosha
<[EMAIL PROTECTED]> wrote:
> A source produces data in kafka at different stages of execution. Topic is partitioned by number of execution stages.
> Partition number shall give us that information.
>
> Eg. 0 specifies data received by source, #last partition specifies processed data delivered by source etc.
>
> Thanks,
> Hussain
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, June 27, 2013 7:47 PM
> To: [EMAIL PROTECTED]
> Subject: Re: How to get partition information from Kafka Stream?
>
> Currently, partition # is not exposed in MessageAndMetaData. We do plan to expose such information in a future release. How do you plan to use this info?
>
> Thanks,
>
> Jun
>
>
> On Thu, Jun 27, 2013 at 7:13 AM, Hussain Pirosha < [EMAIL PROTECTED]> wrote:
>
>> After retrieving a kafka stream or kafka message how to get the
>> corresponding partition number to which it belongs ? I am using kafka
>> version 0.7.2.
>> More specifically kafka.consumer.KafkaStream and
>> kafka.message.MessageAndMetaData classes, does not provide API to
>> retrieve partition number. Are there any other API's to get the partition number ?
>>
>> Thanks,
>> Hussain
>>
>>
>> ________________________________
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited
>> when received in error. Impetus does not represent, warrant and/or
>> guarantee, that the integrity of this communication has been
>> maintained nor that the communication is free of errors, virus, interception or interference.
>>
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.