Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - producer creating multiple log files after message sending


+
Snehalata Nagaje 2013-04-17, 07:06
+
Neha Narkhede 2013-04-17, 15:38
+
Snehalata Nagaje 2013-04-18, 06:11
Copy link to this message
-
RE: producer creating multiple log files after message sending
Snehalata Nagaje 2013-04-18, 08:53
Hi Neha,

I was able to figure this out, this is happening due to time based rolling.

But now problem, when I fetch the messages for particular topic, server is
reading only one log file, returns those messages only.

Is there any way I can combine all log files which are already produced into
one, so that I can get all messages posted in that topic.

Thanks,
Snehalata
-----Original Message-----
From: Snehalata Nagaje [mailto:[EMAIL PROTECTED]]
Sent: Thursday, April 18, 2013 11:41 AM
To: [EMAIL PROTECTED]
Subject: RE: producer creating multiple log files after message sending

Thanks Neha,

-----Original Message-----
From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 17, 2013 9:08 PM
To: [EMAIL PROTECTED]
Subject: Re: producer creating multiple log files after message sending

Hi Snehalata,

Would you grep through the log4j files for the topic under
consideration and look for "Rolling.*due to". This will tell us why
the log is getting rolled over.

Thanks,
Neha

On Wed, Apr 17, 2013 at 12:05 AM, Snehalata Nagaje
<[EMAIL PROTECTED]> wrote:
> Hi,
>
>
>
> I am using kafak 0.8 version and sending messages to kafka server using
> below code,
>
>
>
> Producer<Integer, String> producer = new Producer<Integer, String>(new
> ProducerConfig(props));
>
> producer.send(new KeyedMessage(topicName,partitionKey,jsonString));
>
> producer.close();
>
>
>
> I am able to post messages with, but after certain days, kafka creating
> multiple log files, messages are posted in new log file, although old file
> size is very less.
>
>
>
> What can be the reason for creating multiple log files? Does message
format
> matters while sending message to kafka?
>
>
>
> Thanks,
>
> Snehalata
>
>
> Disclaimer:
> This e-mail may contain Privileged/Confidential information and is
intended only for the individual(s) named. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. Please notify the sender, if you have received this e-mail by
mistake and delete it from your system. Information in this message that
does not relate to the official business of the company shall be understood
as neither given nor endorsed by it. E-mail transmission cannot be
guaranteed to be secure or error-free. The sender does not accept liability
for any errors or omissions in the contents of this message which arise as a
result of e-mail transmission. If verification is required please request a
hard-copy version. Visit us at http://www.harbingergroup.com/

Disclaimer:
This e-mail may contain Privileged/Confidential information and is intended
only for the individual(s) named. Any review, retransmission, dissemination
or other use of, or taking of any action in reliance upon this information
by persons or entities other than the intended recipient is prohibited.
Please notify the sender, if you have received this e-mail by mistake and
delete it from your system. Information in this message that does not relate
to the official business of the company shall be understood as neither given
nor endorsed by it. E-mail transmission cannot be guaranteed to be secure or
error-free. The sender does not accept liability for any errors or omissions
in the contents of this message which arise as a result of e-mail
transmission. If verification is required please request a hard-copy
version. Visit us at http://www.harbingergroup.com/

Disclaimer:
This e-mail may contain Privileged/Confidential information and is intended only for the individual(s) named. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. Please notify the sender, if you have received this e-mail by mistake and delete it from your system. Information in this message that does not relate to the official business of the company shall be understood as neither given nor endorsed by it. E-mail transmission cannot be guaranteed to be secure or error-free. The sender does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Visit us at http://www.harbingergroup.com/

 
+
Jason Huang 2013-04-18, 13:07
+
Neha Narkhede 2013-04-18, 15:26
+
Jason Huang 2013-04-18, 15:37
+
Jay Kreps 2013-04-18, 16:20
+
Jason Huang 2013-04-18, 16:36