Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> producer creating multiple log files after message sending


Copy link to this message
-
Re: producer creating multiple log files after message sending
There is no way to merge those log files in 0.8. But why would you want to
do that ? To reduce the number of open file handles ?

Thanks,
Neha

On Thursday, April 18, 2013, Jason Huang wrote:

> Just curious - given the situation above, if you have two sets of .log and
> .index files under the same topic and same partition, is there any way to
> merge these two sets of log files and index files?
>
>
> On Thu, Apr 18, 2013 at 5:42 AM, Swapnil Ghike <[EMAIL PROTECTED]>
> wrote:
>
> > Otherwise if you are using the high level consumer
> > (ZookeeperConsumerConnector) in code, you will have to start a new
> > consumer with a different group id.
> >
> > Thanks,
> > Swapnil
> >
> > On 4/18/13 2:09 AM, "Swapnil Ghike" <[EMAIL PROTECTED]> wrote:
> >
> > >Snehalata,
> > >
> > >Are you using the console consumer? You can pass a --from-beginning flag
> > >to the console consumer to get all the messages posted to that topic.
> > >
> > >Thanks,
> > >Swapnil
> > >
> > >On 4/18/13 1:32 AM, "Snehalata Nagaje"
> > ><[EMAIL PROTECTED]> wrote:
> > >
> > >>Hi Neha,
> > >>
> > >>I was able to figure this out, this is happening due to time based
> > >>rolling.
> > >>
> > >>But now problem, when I fetch the messages for particular topic, server
> > >>is
> > >>reading only one log file, returns those messages only.
> > >>
> > >>Is there any way I can combine all log files which are already produced
> > >>into
> > >>one, so that I can get all messages posted in that topic.
> > >>
> > >>Thanks,
> > >>Snehalata
> > >>
> > >>
> > >>-----Original Message-----
> > >>From: Snehalata Nagaje [mailto:[EMAIL PROTECTED]]
> > >>Sent: Thursday, April 18, 2013 11:41 AM
> > >>To: [EMAIL PROTECTED]
> > >>Subject: RE: producer creating multiple log files after message sending
> > >>
> > >>Thanks Neha,
> > >>
> > >>-----Original Message-----
> > >>From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> > >>Sent: Wednesday, April 17, 2013 9:08 PM
> > >>To: [EMAIL PROTECTED]
> > >>Subject: Re: producer creating multiple log files after message sending
> > >>
> > >>Hi Snehalata,
> > >>
> > >>Would you grep through the log4j files for the topic under
> > >>consideration and look for "Rolling.*due to". This will tell us why
> > >>the log is getting rolled over.
> > >>
> > >>Thanks,
> > >>Neha
> > >>
> > >>On Wed, Apr 17, 2013 at 12:05 AM, Snehalata Nagaje
> > >><[EMAIL PROTECTED]> wrote:
> > >>> Hi,
> > >>>
> > >>>
> > >>>
> > >>> I am using kafak 0.8 version and sending messages to kafka server
> using
> > >>> below code,
> > >>>
> > >>>
> > >>>
> > >>> Producer<Integer, String> producer = new Producer<Integer,
> String>(new
> > >>> ProducerConfig(props));
> > >>>
> > >>> producer.send(new KeyedMessage(topicName,partitionKey,jsonString));
> > >>>
> > >>> producer.close();
> > >>>
> > >>>
> > >>>
> > >>> I am able to post messages with, but after certain days, kafka
> creating
> > >>> multiple log files, messages are posted in new log file, although old
> > >>>file
> > >>> size is very less.
> > >>>
> > >>>
> > >>>
> > >>> What can be the reason for creating multiple log files? Does message
> > >>format
> > >>> matters while sending message to kafka?
> > >>>
> > >>>
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Snehalata
> > >>>
> > >>>
> > >>> Disclaimer:
> > >>> This e-mail may contain Privileged/Confidential information and is
> > >>intended on

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB