-Re: Hadoop-consumer & partition question
Felix GV 2012-08-13, 19:08
I haven't used this script in a while, but if I remember correctly, you
should have a different offset file for each broker/partition combination...
In any case, the article you linked to is an outdated version of that
script (as mentioned in the block at the very beginning of the post, BTW).
A quick look at the script you linked to shows that it manages only a
single offset file, which would explain why you're consuming just one
partition. The latest version of the script manages all of the offset
files, so I think that should solve your problem.
You can find the latest version of the script here:
On Mon, Aug 13, 2012 at 2:39 AM, Fredrik Emilsson <
[EMAIL PROTECTED]> wrote:
> Thanks for the information!
> Here is my offset file:
> device-events-2 0 5133247
> Is the 0 in the file the partition? If so then I have to identify how to
> set the partition number. I guess 0 is the default value.
> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: 10 August 2012 17:34
> To: [EMAIL PROTECTED]
> Subject: Re: Hadoop-consumer & partition question
> The Hadoop consumer uses an offsets file to know which partitions to
> consume and from which offset. How does you offsets file look like ?
> On Fri, Aug 10, 2012 at 7:06 AM, Fredrik Emilsson <
> [EMAIL PROTECTED]> wrote:
> > Hello,
> > I have a topic that have two partitions. I have one broker. Adding
> > events is working well, there are events both in the 0 and 1 partition.
> > The problem arise when I try to import it into Hadoop. It seems that
> > only the events in partition 0 is imported. I am using a script (found
> > here:
> > http://felixgv.com/post/69/automating-incremental-imports-with-the-kaf
> > ka
> > -hadoop-consumer/) to import it.
> > Does anyone know what the problem could be?
> > Regards,
> > Fredrik
> > NOTICE - This message and any attached files may contain information
> that is confidential and/or subject of legal privilege intended only for
> use by the intended recipient. If you are not the intended recipient or the
> person responsible for delivering the message to the intended recipient, be
> advised that you have received this message in error and that any
> dissemination, copying or use of this message or attachment is strictly
> forbidden, as is the disclosure of the information therein. If you have
> received this message in error please notify the sender immediately and
> delete the message.