Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Overzealous logging?


+
Jason Rosenberg 2013-09-23, 22:27
+
Neha Narkhede 2013-09-24, 01:51
+
Jason Rosenberg 2013-09-24, 16:01
+
Neha Narkhede 2013-09-24, 16:05
Copy link to this message
-
Re: Overzealous logging?
Jason Rosenberg 2013-09-24, 18:25
filed: https://issues.apache.org/jira/browse/KAFKA-1066
On Tue, Sep 24, 2013 at 12:04 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> This makes sense. Please file a JIRA where we can discuss a patch.
>
> Thanks,
> Neha
>
>
> On Tue, Sep 24, 2013 at 9:00 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > I'm wondering if a simple change could be to not log full stack traces
> for
> > simple things like "Connection refused", etc.  Seems it would be fine to
> > just log the exception message in such cases.
> >
> > Also, the log levels could be tuned, such that things logged as ERROR
> > indicate that all possible retries have been attempted, rather than
> having
> > it be an ERROR for each step of the retry/failover process.  Thus, for a
> > redundant, clustered service, it should be considered normal that single
> > nodes will be unavailable (such as when we're doing a rolling restart of
> > the cluster, etc.).  It should only be an ERROR if all brokers/all
> replicas
> > are unavailable, etc.  This way, we can selectively set our log level to
> > ERROR, and have it be useful.
> >
> > Does this make sense?  If so, I can file a Jira along these lines....
> >
> > Jason
> >
> >
> > On Mon, Sep 23, 2013 at 9:51 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Agree that this is annoying. We plan to fix this largely as part of the
> > > Client
> > > Rewrite <
> > https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite
> > > >project.
> > > If you have ideas on fixing it before that, could you file a JIRA
> > > where we can start a discussion?
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Mon, Sep 23, 2013 at 3:26 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Sorry for the crazy long log trace here (feel free to ignore this
> > message
> > > > :))
> > > >
> > > > I'm just wondering if there's an easy way to sensibly reduce the
> amount
> > > of
> > > > logging that a kafka produer (0.8) will emit if I try to send a
> message
> > > > (with ack level 1), if no broker is currently running?
> > > >
> > > > This is from one of my unit tests.  I am using the default retry
> count
> > > (3)
> > > > here, but even if I reduced that, it seems this is a crazy amount of
> > > > logging.  (I've edited this to remove from each stack trace the
> portion
> > > of
> > > > testing/calling code, into the Producer.send() call).
> > > >
> > > > Here's the code snippet that produced the logging below (and note,
> the
> > > > server was not available on the requested port).
> > > >
> > > >       KeyedMessage<Integer, T> msg = new KeyedMessage<Integer,
> > T>(topic,
> > > > message);
> > > >       producer.send(msg);
> > > >
> > > > Jason
> > > >
> > > >
> > > > 599 [main] ERROR kafka.producer.SyncProducer  - Producer connection
> to
> > > > localhost:1025 unsuccessful
> > > > java.net.ConnectException: Connection refused
> > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:639)
> > > > at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > at
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > > > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > > > at
> > > >
> > >
> >
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:69)
> > > > at kafka.utils.Utils$.swallow(Utils.scala:186)
> > > > at kafka.utils.Logging$class.swallowError(Logging.scala:105)