We probably need to add a small amount of logging in the new producer and (soon) consumer clients. I wanted to have a quick discussion on logging libraries before I start adding this in the producer.
Previously we have been pretty verbose loggers and I think we should stop that. For clients you mostly don't need to log: if there is an error you should throw it back not log it, so you don't need ERROR logging. Likewise I think it is rude to pollute peoples logs with the details of client initialization (establishing connections, etc), so you don't need INFO logging. However perhaps there is an argument to be made for WARN and DEBUG. I think it is perhaps useful to log a WARN when a server breaks a connection or metadata initialization fails. It can sometimes also be useful to be able to enable debug logging to see step by step processing in the client, which is the case for DEBUG logging.
Here is my knowledge about the state of Java logging: 1. Most people still use log4j. The server is using log4j. 2. Second runner-up in slf4j. I personally consider slf4j pretty silly but this is perhaps the more flexible choice since people can plug in different stuff. 3. java.util.logging ships with the jdk, but for some reason no one uses it. 4. There is no critical mass around any other logging library.
The context for how to think about this is the following. We are not trying to pick the "best" logging library. Fundamentally logging is pretty straight-forward and for our simple use case it is inconceivable that any particular library could be much better than any other in terms of feature set. We want the most standard library. My goal is to minimize the dependencies of the client and make our basic logging cases work for most cases.
Is there a reason not to just the java.util.logging? It comes with the jdk and supports pluggable appenders so people who have some other library can plug that in right?
Basically my preference would be java.util.logging unless there is some known problem with it, otherwise I guess slf4j, and if not that then log4j.
We are already using other libraries in various parts of our code (e.g., metrics, zkclient, joptsimple, etc) some of which pull in these other logging dependencies anyway. i.e., what do we gain by using jul? There may be a good reason why people don't use jul so I think we should fully understand that before going with jul. So it may be simpler to just stick with log4j for the client rewrites and investigate logging later.
log4j2 is becoming more widespread and many users seem to be favorable toward logback. slf4j would cover all of these very easily. From what I understand jul does not make it very easy to plug in with these various options but I could be wrong.
I completely agree on the need to fix our client logging as that will go a long way in usability for end-users unless we want to keep getting asked the "Why do I see this ERROR in my logs..?" questions.
On Mon, Feb 03, 2014 at 11:08:39AM -0800, Neha Narkhede wrote:
My vote would be with log4j, I don't have that much experience with log4j2 or a good feel for how much the industry is moving towards it. On Mon, Feb 3, 2014 at 11:17 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
I see server and client logging and dependencies are really different in my mind. The server is a standalone process. Once we decouple the clients I think we can actually be a lot more aggressive about using more libraries on the server, why not? We have avoided that so far because they inevitably leak into the clients since they arent separated.
The problem on the client side is that if we choose anything used by our clients and there is any incompatibility then they can't use kafka.
From my point of view the improved logging libraries (log4j2, logback, etc) are the worst option. They are new so likely will have api changes, and few people use them so they don't get the benefit of sharing the same logging library.
-Jay On Mon, Feb 3, 2014 at 11:17 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
+1 for SLF4J. It is the only option that doesn't force a consumer to use a particular style of configuration. It is common enough that I think it would not cause any adoption friction. There's a relevant and lengthier discussion here: http://stackoverflow.com/questions/11359187/dont-get-it-why-not-jul On Mon, Feb 3, 2014 at 3:00 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
Right - I was actually advocating sl4fj for this very reason (not a specific logging implementation).
Clients would use the producer/consumer as part of some larger system that is more than likely to use something other than j.u.l (since no one seems to use it). With a layer like slf4j clients can choose whatever logging implementation they wish. If we go with j.u.l then we would force all clients to include a j.u.l configuration file just for their producer/consumer in addition to the configuration file for the rest of their system.
-Jay On Mon, Feb 3, 2014 at 1:11 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext