Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Overzealous logging?


+
Jason Rosenberg 2013-09-23, 22:27
+
Neha Narkhede 2013-09-24, 01:51
+
Jason Rosenberg 2013-09-24, 16:01
+
Neha Narkhede 2013-09-24, 16:05
Copy link to this message
-
Re: Overzealous logging?
filed: https://issues.apache.org/jira/browse/KAFKA-1066
On Tue, Sep 24, 2013 at 12:04 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> This makes sense. Please file a JIRA where we can discuss a patch.
>
> Thanks,
> Neha
>
>
> On Tue, Sep 24, 2013 at 9:00 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > I'm wondering if a simple change could be to not log full stack traces
> for
> > simple things like "Connection refused", etc.  Seems it would be fine to
> > just log the exception message in such cases.
> >
> > Also, the log levels could be tuned, such that things logged as ERROR
> > indicate that all possible retries have been attempted, rather than
> having
> > it be an ERROR for each step of the retry/failover process.  Thus, for a
> > redundant, clustered service, it should be considered normal that single
> > nodes will be unavailable (such as when we're doing a rolling restart of
> > the cluster, etc.).  It should only be an ERROR if all brokers/all
> replicas
> > are unavailable, etc.  This way, we can selectively set our log level to
> > ERROR, and have it be useful.
> >
> > Does this make sense?  If so, I can file a Jira along these lines....
> >
> > Jason
> >
> >
> > On Mon, Sep 23, 2013 at 9:51 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Agree that this is annoying. We plan to fix this largely as part of the
> > > Client
> > > Rewrite <
> > https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite
> > > >project.
> > > If you have ideas on fixing it before that, could you file a JIRA
> > > where we can start a discussion?
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Mon, Sep 23, 2013 at 3:26 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Sorry for the crazy long log trace here (feel free to ignore this
> > message
> > > > :))
> > > >
> > > > I'm just wondering if there's an easy way to sensibly reduce the
> amount
> > > of
> > > > logging that a kafka produer (0.8) will emit if I try to send a
> message
> > > > (with ack level 1), if no broker is currently running?
> > > >
> > > > This is from one of my unit tests.  I am using the default retry
> count
> > > (3)
> > > > here, but even if I reduced that, it seems this is a crazy amount of
> > > > logging.  (I've edited this to remove from each stack trace the
> portion
> > > of
> > > > testing/calling code, into the Producer.send() call).
> > > >
> > > > Here's the code snippet that produced the logging below (and note,
> the
> > > > server was not available on the requested port).
> > > >
> > > >       KeyedMessage<Integer, T> msg = new KeyedMessage<Integer,
> > T>(topic,
> > > > message);
> > > >       producer.send(msg);
> > > >
> > > > Jason
> > > >
> > > >
> > > > 599 [main] ERROR kafka.producer.SyncProducer  - Producer connection
> to
> > > > localhost:1025 unsuccessful
> > > > java.net.ConnectException: Connection refused
> > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:639)
> > > > at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > at
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > > > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > > > at
> > > >
> > >
> >
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
> > > > at
> > > >
> > > >
> > >
> >
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:69)
> > > > at kafka.utils.Utils$.swallow(Utils.scala:186)
> > > > at kafka.utils.Logging$class.swallowError(Logging.scala:105)

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB