Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> preparing for the 0.8 final release


Copy link to this message
-
Re: preparing for the 0.8 final release
Ok,

So here's an outline of what I think seems to have happened.

I have a consumer, that uses a filter to consume a large number of topics
(e.g. several hundred).  Each topic has only a single partition.

It normally has no trouble keeping up processing all messages on all
topics.   However, we had a case a couple days ago where it seemed to hang,
and not consume anything for several hours.  I restarted the consumer (and
now I've updated it from 0.8-beta1 to 0.8-latest-HEAD).  Data is flowing
again, but some topics are seeming to take much longer than others to catch
up.  The slow ones seem to be the topics that have more data than others (a
loose theory at present).

Does that make sense?  If I understand things correctly, the consumer will
fetch chunks of data from each topic/partition, in order, in a big loop?
 So if it has caught up with most of the topics, will it waste time
re-polling all those (and getting nothing) before coming back to the topics
that are lagging?  Perhaps having a larger fetch size would help here?

Jason
On Sat, Oct 19, 2013 at 6:24 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> I'll try to, next time it hangs!
>
>
> On Sat, Oct 19, 2013 at 4:04 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> Can you send around a thread dump of the halted consumer process?
>>
>>
>>
>> On Sat, Oct 19, 2013 at 12:16 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> wrote:
>>
>> > The latest HEAD does seem to solve one issue, where a new topic being
>> > created after the consumer is started, would not be consumed.
>> >
>> > But the bigger issue is that we have a couple different consumers both
>> > consuming the same set of topics (under different groupids), and hanging
>> > after a while (both hanging at about the same point).  The topics in
>> each
>> > case are selected with a filter (actually a relatively large number of
>> > topics, some of which are newly created over time).  I'm still not sure
>> > whether the new version is solving this issue (since it was a rare
>> > transient thing anyway).
>> >
>> > Thanks,
>> >
>> > Jason
>> >
>> >
>> > On Sat, Oct 19, 2013 at 2:03 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> >
>> > > Yes, 0.8 will be released from the HEAD of the 0.8 branch. Is the
>> problem
>> > > with consuming new topics or topics whose partitions are increased? If
>> > so,
>> > > see KAFKA-1030 and KAFKA-1075.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > > On Fri, Oct 18, 2013 at 4:03 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> > wrote:
>> > >
>> > > > Will the 0.8 release come from the HEAD of the 0.8 branch?  I'd
>> like to
>> > > > experiment with it, to see if it solves some of the issues I'm
>> seeing,
>> > > with
>> > > > consumers refusing to consume new messages.  We've been using the
>> beta1
>> > > > version.
>> > > >
>> > > > I remember mention there was a Jira issues along these lines, which
>> was
>> > > > fixed post 0.8-beta1.  Which issue was that (I'd like to see if it
>> > > matches
>> > > > what I'm seeing).
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Jason
>> > > >
>> > > >
>> > > > On Wed, Oct 9, 2013 at 8:04 PM, Jay Kreps <[EMAIL PROTECTED]>
>> wrote:
>> > > >
>> > > > > I uploaded a patch against trunk which also fixes KAFKA-1036, the
>> > other
>> > > > > knows windows issue. Review appreciated. Should be an easy one.
>> > > > >
>> > > > > https://issues.apache.org/jira/browse/KAFKA-1008
>> > > > >
>> > > > > -Jay
>> > > > >
>> > > > >
>> > > > > On Wed, Oct 9, 2013 at 8:56 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> > > > >
>> > > > > > KAFKA-1008 has been checked into the 0.8 branch and needs to be
>> > > > manually
>> > > > > > double-committed to trunk. To avoid merging problems, I suggest
>> > that
>> > > > for
>> > > > > > all future changes in the 0.8 branch, we double commit them to
>> > trunk.
>> > > > Any
>> > > > > > objections?
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Jun
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Oct 7, 2013 at 5:33 PM, Jun Rao <[EMAIL PROTECTED]>