Not sure what's the causing the slow down. Is the second fetch geting more
data?

Also, if all those partitions are on the same broker, you could have
fetched all partitions' data in a single fetch request.

Thanks,

Jun

On Mon, Jan 28, 2013 at 6:02 AM, Danny Yeshurun <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I am using Kafka 0.7.2
> I have a consumer that tries to fetch messages from several partitions on
> the same broker (and topic).
> The consumers issues a fetch request per partition, if there are messages
> to read, it fetches the first one and goes to the next partition to fetch
> another message and so on (in a loop).
> When there are not messages in a partition, the consumer will issue another
> fetch request on the next cycle etc.
> I noticed that starting with the 2nd fetch request, response time becomes
> unacceptable, i.e. reading back the response from the broker takes ~0.04
> second (while it took ~0.0006 sec on the first time).
> This behavior is consistent over different runs.
>
> Any idea?
>
> Thanks,
> Danny
>

 

No data on the second time. The partition is empty but starting from the
second request the response is slow.

On Mon, Jan 28, 2013 at 6:31 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Not sure what's the causing the slow down. Is the second fetch geting more
> data?
>
> Also, if all those partitions are on the same broker, you could have
> fetched all partitions' data in a single fetch request.
>
> Thanks,
>
> Jun
>
> On Mon, Jan 28, 2013 at 6:02 AM, Danny Yeshurun <[EMAIL PROTECTED]
> >wrote:
>
> > Hi,
> >
> > I am using Kafka 0.7.2
> > I have a consumer that tries to fetch messages from several partitions on
> > the same broker (and topic).
> > The consumers issues a fetch request per partition, if there are messages
> > to read, it fetches the first one and goes to the next partition to fetch
> > another message and so on (in a loop).
> > When there are not messages in a partition, the consumer will issue
> another
> > fetch request on the next cycle etc.
> > I noticed that starting with the 2nd fetch request, response time becomes
> > unacceptable, i.e. reading back the response from the broker takes ~0.04
> > second (while it took ~0.0006 sec on the first time).
> > This behavior is consistent over different runs.
> >
> > Any idea?
> >
> > Thanks,
> > Danny
> >
>

 

Danny,

Please can you file a bug to explain the issue ?

Thanks,
Neha
On Mon, Jan 28, 2013 at 8:57 AM, Danny Yeshurun <[EMAIL PROTECTED]>wrote:

> No data on the second time. The partition is empty but starting from the
> second request the response is slow.
>
> On Mon, Jan 28, 2013 at 6:31 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Not sure what's the causing the slow down. Is the second fetch geting
> more
> > data?
> >
> > Also, if all those partitions are on the same broker, you could have
> > fetched all partitions' data in a single fetch request.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 28, 2013 at 6:02 AM, Danny Yeshurun <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi,
> > >
> > > I am using Kafka 0.7.2
> > > I have a consumer that tries to fetch messages from several partitions
> on
> > > the same broker (and topic).
> > > The consumers issues a fetch request per partition, if there are
> messages
> > > to read, it fetches the first one and goes to the next partition to
> fetch
> > > another message and so on (in a loop).
> > > When there are not messages in a partition, the consumer will issue
> > another
> > > fetch request on the next cycle etc.
> > > I noticed that starting with the 2nd fetch request, response time
> becomes
> > > unacceptable, i.e. reading back the response from the broker takes
> ~0.04
> > > second (while it took ~0.0006 sec on the first time).
> > > This behavior is consistent over different runs.
> > >
> > > Any idea?
> > >
> > > Thanks,
> > > Danny
> > >
> >
>

 

Will do.

Regarding the multi-fetch request - say I give a list of topic-partition,
is it guaranteed that all pairs will be "scanned", i.e. if there is a
message in a partition I will always get it in one call?

Thanks,
Danny

On Mon, Jan 28, 2013 at 7:54 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Danny,
>
> Please can you file a bug to explain the issue ?
>
> Thanks,
> Neha
>
>
> On Mon, Jan 28, 2013 at 8:57 AM, Danny Yeshurun <[EMAIL PROTECTED]
> >wrote:
>
> > No data on the second time. The partition is empty but starting from the
> > second request the response is slow.
> >
> > On Mon, Jan 28, 2013 at 6:31 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Not sure what's the causing the slow down. Is the second fetch geting
> > more
> > > data?
> > >
> > > Also, if all those partitions are on the same broker, you could have
> > > fetched all partitions' data in a single fetch request.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Jan 28, 2013 at 6:02 AM, Danny Yeshurun <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > I am using Kafka 0.7.2
> > > > I have a consumer that tries to fetch messages from several
> partitions
> > on
> > > > the same broker (and topic).
> > > > The consumers issues a fetch request per partition, if there are
> > messages
> > > > to read, it fetches the first one and goes to the next partition to
> > fetch
> > > > another message and so on (in a loop).
> > > > When there are not messages in a partition, the consumer will issue
> > > another
> > > > fetch request on the next cycle etc.
> > > > I noticed that starting with the 2nd fetch request, response time
> > becomes
> > > > unacceptable, i.e. reading back the response from the broker takes
> > ~0.04
> > > > second (while it took ~0.0006 sec on the first time).
> > > > This behavior is consistent over different runs.
> > > >
> > > > Any idea?
> > > >
> > > > Thanks,
> > > > Danny
> > > >
> > >
> >
>

 

> if there is a
> message in a partition I will always get it in one call?
>

Yes, if you are talking to the right broker, it will fetch all the
available data for all partitions in the multi fetch request.

Thanks,
Neha

 

https://issues.apache.org/jira/browse/KAFKA-744

On Tue, Jan 29, 2013 at 6:31 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> > if there is a
> > message in a partition I will always get it in one call?
> >
>
> Yes, if you are talking to the right broker, it will fetch all the
> available data for all partitions in the multi fetch request.
>
> Thanks,
> Neha
>