Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> producer creating multiple log files after message sending


Copy link to this message
-
Re: producer creating multiple log files after message sending
There is no way to merge those log files in 0.8. But why would you want to
do that ? To reduce the number of open file handles ?

Thanks,
Neha

On Thursday, April 18, 2013, Jason Huang wrote:

> Just curious - given the situation above, if you have two sets of .log and
> .index files under the same topic and same partition, is there any way to
> merge these two sets of log files and index files?
>
>
> On Thu, Apr 18, 2013 at 5:42 AM, Swapnil Ghike <[EMAIL PROTECTED]>
> wrote:
>
> > Otherwise if you are using the high level consumer
> > (ZookeeperConsumerConnector) in code, you will have to start a new
> > consumer with a different group id.
> >
> > Thanks,
> > Swapnil
> >
> > On 4/18/13 2:09 AM, "Swapnil Ghike" <[EMAIL PROTECTED]> wrote:
> >
> > >Snehalata,
> > >
> > >Are you using the console consumer? You can pass a --from-beginning flag
> > >to the console consumer to get all the messages posted to that topic.
> > >
> > >Thanks,
> > >Swapnil
> > >
> > >On 4/18/13 1:32 AM, "Snehalata Nagaje"
> > ><[EMAIL PROTECTED]> wrote:
> > >
> > >>Hi Neha,
> > >>
> > >>I was able to figure this out, this is happening due to time based
> > >>rolling.
> > >>
> > >>But now problem, when I fetch the messages for particular topic, server
> > >>is
> > >>reading only one log file, returns those messages only.
> > >>
> > >>Is there any way I can combine all log files which are already produced
> > >>into
> > >>one, so that I can get all messages posted in that topic.
> > >>
> > >>Thanks,
> > >>Snehalata
> > >>
> > >>
> > >>-----Original Message-----
> > >>From: Snehalata Nagaje [mailto:[EMAIL PROTECTED]]
> > >>Sent: Thursday, April 18, 2013 11:41 AM
> > >>To: [EMAIL PROTECTED]
> > >>Subject: RE: producer creating multiple log files after message sending
> > >>
> > >>Thanks Neha,
> > >>
> > >>-----Original Message-----
> > >>From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > >>Sent: Wednesday, April 17, 2013 9:08 PM
> > >>To: [EMAIL PROTECTED]
> > >>Subject: Re: producer creating multiple log files after message sending
> > >>
> > >>Hi Snehalata,
> > >>
> > >>Would you grep through the log4j files for the topic under
> > >>consideration and look for "Rolling.*due to". This will tell us why
> > >>the log is getting rolled over.
> > >>
> > >>Thanks,
> > >>Neha
> > >>
> > >>On Wed, Apr 17, 2013 at 12:05 AM, Snehalata Nagaje
> > >><[EMAIL PROTECTED]> wrote:
> > >>> Hi,
> > >>>
> > >>>
> > >>>
> > >>> I am using kafak 0.8 version and sending messages to kafka server
> using
> > >>> below code,
> > >>>
> > >>>
> > >>>
> > >>> Producer<Integer, String> producer = new Producer<Integer,
> String>(new
> > >>> ProducerConfig(props));
> > >>>
> > >>> producer.send(new KeyedMessage(topicName,partitionKey,jsonString));
> > >>>
> > >>> producer.close();
> > >>>
> > >>>
> > >>>
> > >>> I am able to post messages with, but after certain days, kafka
> creating
> > >>> multiple log files, messages are posted in new log file, although old
> > >>>file
> > >>> size is very less.
> > >>>
> > >>>
> > >>>
> > >>> What can be the reason for creating multiple log files? Does message
> > >>format
> > >>> matters while sending message to kafka?
> > >>>
> > >>>
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Snehalata
> > >>>
> > >>>
> > >>> Disclaimer:
> > >>> This e-mail may contain Privileged/Confidential information and is
> > >>intended on