Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> producer creating multiple log files after message sending


Copy link to this message
-
Re: producer creating multiple log files after message sending
Jay,

This makes perfect sense and it is very helpful.

thanks,

Jason
On Thu, Apr 18, 2013 at 12:20 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> I think there is a misunderstanding here. The log file division is
> completely transparent to the consumer. The consumer consumes by
> offset (0,1,2,3,...) and these offsets span log files (i.e. there is a
> single, monotonically increasing set of offsets over all log files in
> a partition). In other words consumers ALWAYS consume across log files
> based on the position (offset) in the set of all logs in the
> partition.
>
> If you want fewer log files change the settings so that the log files
> are bigger. However the only impact of this is fewer file handles, it
> won't impact the consumer in any way at all.
>
> Not sure if that helps...
> -Jay
>
> On Thu, Apr 18, 2013 at 8:36 AM, Jason Huang <[EMAIL PROTECTED]>
> wrote:
> > Number of open file handles is one thing. But I am thinking more of a use
> > case where you may want to fetch information from two separate log files
> of
> > same topic and same partition. Could we use one high level consumer API
> > call to achieve that? One naive idea is that if we can merge those two
> log
> > files (since they are of same topic and in partition I thought we might
> be
> > able to merge them) into one, then using one high level consumer API will
> > do the job.
> >
> >
> > On Thu, Apr 18, 2013 at 11:26 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
> >
> >> There is no way to merge those log files in 0.8. But why would you want
> to
> >> do that ? To reduce the number of open file handles ?
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Thursday, April 18, 2013, Jason Huang wrote:
> >>
> >> > Just curious - given the situation above, if you have two sets of .log
> >> and
> >> > .index files under the same topic and same partition, is there any
> way to
> >> > merge these two sets of log files and index files?
> >> >
> >> >
> >> > On Thu, Apr 18, 2013 at 5:42 AM, Swapnil Ghike <[EMAIL PROTECTED]>
> >> > wrote:
> >> >
> >> > > Otherwise if you are using the high level consumer
> >> > > (ZookeeperConsumerConnector) in code, you will have to start a new
> >> > > consumer with a different group id.
> >> > >
> >> > > Thanks,
> >> > > Swapnil
> >> > >
> >> > > On 4/18/13 2:09 AM, "Swapnil Ghike" <[EMAIL PROTECTED]> wrote:
> >> > >
> >> > > >Snehalata,
> >> > > >
> >> > > >Are you using the console consumer? You can pass a --from-beginning
> >> flag
> >> > > >to the console consumer to get all the messages posted to that
> topic.
> >> > > >
> >> > > >Thanks,
> >> > > >Swapnil
> >> > > >
> >> > > >On 4/18/13 1:32 AM, "Snehalata Nagaje"
> >> > > ><[EMAIL PROTECTED]> wrote:
> >> > > >
> >> > > >>Hi Neha,
> >> > > >>
> >> > > >>I was able to figure this out, this is happening due to time based
> >> > > >>rolling.
> >> > > >>
> >> > > >>But now problem, when I fetch the messages for particular topic,
> >> server
> >> > > >>is
> >> > > >>reading only one log file, returns those messages only.
> >> > > >>
> >> > > >>Is there any way I can combine all log files which are already
> >> produced
> >> > > >>into
> >> > > >>one, so that I can get all messages posted in that topic.
> >> > > >>
> >> > > >>Thanks,
> >> > > >>Snehalata
> >> > > >>
> >> > > >>
> >> > > >>-----Original Message-----
> >> > > >>From: Snehalata Nagaje [mailto:
> [EMAIL PROTECTED]]
> >> > > >>Sent: Thursday, April 18, 2013 11:41 AM
> >> > > >>To: [EMAIL PROTECTED]
> >> > > >>Subject: RE: producer creating multiple log files after message
> >> sending
> >> > > >>
> >> > > >>Thanks Neha,
> >> > > >>
> >> > > >>-----Original Message-----
> >> > > >>From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> >> > > >>Sent: Wednesday, April 17, 2013 9:08 PM
> >> > > >>To: [EMAIL PROTECTED]
> >> > > >>Subject: Re: producer creating multiple log files after message
> >> sending
> >> > > >>
> >> > > >>Hi Snehalata,
> >> > > >>
> >> > > >>Would you grep through the log4j files for the topic under

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB