Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> producer creating multiple log files after message sending


+
Snehalata Nagaje 2013-04-17, 07:06
+
Neha Narkhede 2013-04-17, 15:38
+
Snehalata Nagaje 2013-04-18, 06:11
+
Snehalata Nagaje 2013-04-18, 08:53
+
Jason Huang 2013-04-18, 13:07
+
Neha Narkhede 2013-04-18, 15:26
+
Jason Huang 2013-04-18, 15:37
+
Jay Kreps 2013-04-18, 16:20
Copy link to this message
-
Re: producer creating multiple log files after message sending
Jay,

This makes perfect sense and it is very helpful.

thanks,

Jason
On Thu, Apr 18, 2013 at 12:20 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> I think there is a misunderstanding here. The log file division is
> completely transparent to the consumer. The consumer consumes by
> offset (0,1,2,3,...) and these offsets span log files (i.e. there is a
> single, monotonically increasing set of offsets over all log files in
> a partition). In other words consumers ALWAYS consume across log files
> based on the position (offset) in the set of all logs in the
> partition.
>
> If you want fewer log files change the settings so that the log files
> are bigger. However the only impact of this is fewer file handles, it
> won't impact the consumer in any way at all.
>
> Not sure if that helps...
> -Jay
>
> On Thu, Apr 18, 2013 at 8:36 AM, Jason Huang <[EMAIL PROTECTED]>
> wrote:
> > Number of open file handles is one thing. But I am thinking more of a use
> > case where you may want to fetch information from two separate log files
> of
> > same topic and same partition. Could we use one high level consumer API
> > call to achieve that? One naive idea is that if we can merge those two
> log
> > files (since they are of same topic and in partition I thought we might
> be
> > able to merge them) into one, then using one high level consumer API will
> > do the job.
> >
> >
> > On Thu, Apr 18, 2013 at 11:26 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
> >
> >> There is no way to merge those log files in 0.8. But why would you want
> to
> >> do that ? To reduce the number of open file handles ?
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Thursday, April 18, 2013, Jason Huang wrote:
> >>
> >> > Just curious - given the situation above, if you have two sets of .log
> >> and
> >> > .index files under the same topic and same partition, is there any
> way to
> >> > merge these two sets of log files and index files?
> >> >
> >> >
> >> > On Thu, Apr 18, 2013 at 5:42 AM, Swapnil Ghike <[EMAIL PROTECTED]>
> >> > wrote:
> >> >
> >> > > Otherwise if you are using the high level consumer
> >> > > (ZookeeperConsumerConnector) in code, you will have to start a new
> >> > > consumer with a different group id.
> >> > >
> >> > > Thanks,
> >> > > Swapnil
> >> > >
> >> > > On 4/18/13 2:09 AM, "Swapnil Ghike" <[EMAIL PROTECTED]> wrote:
> >> > >
> >> > > >Snehalata,
> >> > > >
> >> > > >Are you using the console consumer? You can pass a --from-beginning
> >> flag
> >> > > >to the console consumer to get all the messages posted to that
> topic.
> >> > > >
> >> > > >Thanks,
> >> > > >Swapnil
> >> > > >
> >> > > >On 4/18/13 1:32 AM, "Snehalata Nagaje"
> >> > > ><[EMAIL PROTECTED]> wrote:
> >> > > >
> >> > > >>Hi Neha,
> >> > > >>
> >> > > >>I was able to figure this out, this is happening due to time based
> >> > > >>rolling.
> >> > > >>
> >> > > >>But now problem, when I fetch the messages for particular topic,
> >> server
> >> > > >>is
> >> > > >>reading only one log file, returns those messages only.
> >> > > >>
> >> > > >>Is there any way I can combine all log files which are already
> >> produced
> >> > > >>into
> >> > > >>one, so that I can get all messages posted in that topic.
> >> > > >>
> >> > > >>Thanks,
> >> > > >>Snehalata
> >> > > >>
> >> > > >>
> >> > > >>-----Original Message-----
> >> > > >>From: Snehalata Nagaje [mailto:
> [EMAIL PROTECTED]]
> >> > > >>Sent: Thursday, April 18, 2013 11:41 AM
> >> > > >>To: [EMAIL PROTECTED]
> >> > > >>Subject: RE: producer creating multiple log files after message
> >> sending
> >> > > >>
> >> > > >>Thanks Neha,
> >> > > >>
> >> > > >>-----Original Message-----
> >> > > >>From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> >> > > >>Sent: Wednesday, April 17, 2013 9:08 PM
> >> > > >>To: [EMAIL PROTECTED]
> >> > > >>Subject: Re: producer creating multiple log files after message
> >> sending
> >> > > >>
> >> > > >>Hi Snehalata,
> >> > > >>
> >> > > >>Would you grep through the log4j files for the topic under