Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> preparing for the 0.8 final release


Copy link to this message
-
Re: preparing for the 0.8 final release
The fetch thread uses multi-fetch. Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whymessagesaredelayedinmyconsumer%3F,
which may be related to your issue.

Thanks,

Jun
On Sun, Oct 20, 2013 at 2:48 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Ok,
>
> So here's an outline of what I think seems to have happened.
>
> I have a consumer, that uses a filter to consume a large number of topics
> (e.g. several hundred).  Each topic has only a single partition.
>
> It normally has no trouble keeping up processing all messages on all
> topics.   However, we had a case a couple days ago where it seemed to hang,
> and not consume anything for several hours.  I restarted the consumer (and
> now I've updated it from 0.8-beta1 to 0.8-latest-HEAD).  Data is flowing
> again, but some topics are seeming to take much longer than others to catch
> up.  The slow ones seem to be the topics that have more data than others (a
> loose theory at present).
>
> Does that make sense?  If I understand things correctly, the consumer will
> fetch chunks of data from each topic/partition, in order, in a big loop?
>  So if it has caught up with most of the topics, will it waste time
> re-polling all those (and getting nothing) before coming back to the topics
> that are lagging?  Perhaps having a larger fetch size would help here?
>
> Jason
>
>
> On Sat, Oct 19, 2013 at 6:24 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > I'll try to, next time it hangs!
> >
> >
> > On Sat, Oct 19, 2013 at 4:04 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
> >
> >> Can you send around a thread dump of the halted consumer process?
> >>
> >>
> >>
> >> On Sat, Oct 19, 2013 at 12:16 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >> > The latest HEAD does seem to solve one issue, where a new topic being
> >> > created after the consumer is started, would not be consumed.
> >> >
> >> > But the bigger issue is that we have a couple different consumers both
> >> > consuming the same set of topics (under different groupids), and
> hanging
> >> > after a while (both hanging at about the same point).  The topics in
> >> each
> >> > case are selected with a filter (actually a relatively large number of
> >> > topics, some of which are newly created over time).  I'm still not
> sure
> >> > whether the new version is solving this issue (since it was a rare
> >> > transient thing anyway).
> >> >
> >> > Thanks,
> >> >
> >> > Jason
> >> >
> >> >
> >> > On Sat, Oct 19, 2013 at 2:03 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >> >
> >> > > Yes, 0.8 will be released from the HEAD of the 0.8 branch. Is the
> >> problem
> >> > > with consuming new topics or topics whose partitions are increased?
> If
> >> > so,
> >> > > see KAFKA-1030 and KAFKA-1075.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun
> >> > >
> >> > >
> >> > > On Fri, Oct 18, 2013 at 4:03 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >> > wrote:
> >> > >
> >> > > > Will the 0.8 release come from the HEAD of the 0.8 branch?  I'd
> >> like to
> >> > > > experiment with it, to see if it solves some of the issues I'm
> >> seeing,
> >> > > with
> >> > > > consumers refusing to consume new messages.  We've been using the
> >> beta1
> >> > > > version.
> >> > > >
> >> > > > I remember mention there was a Jira issues along these lines,
> which
> >> was
> >> > > > fixed post 0.8-beta1.  Which issue was that (I'd like to see if it
> >> > > matches
> >> > > > what I'm seeing).
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Jason
> >> > > >
> >> > > >
> >> > > > On Wed, Oct 9, 2013 at 8:04 PM, Jay Kreps <[EMAIL PROTECTED]>
> >> wrote:
> >> > > >
> >> > > > > I uploaded a patch against trunk which also fixes KAFKA-1036,
> the
> >> > other
> >> > > > > knows windows issue. Review appreciated. Should be an easy one.
> >> > > > >
> >> > > > > https://issues.apache.org/jira/browse/KAFKA-1008
> >> > > > >
> >> > > > > -Jay
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Oct 9, 2013 at 8:56 AM, Jun Rao <[EMAIL PROTECTED]>