How do I intercept Kakfa broker operation so that features such as security,logging,etc can be implemented as a pluggable filter. For example we have "BrokerFilter" class in ActiveMQ , Is there anything similar in Kafka?
Our goal is to provide the best implementation of a set of useful abstractions and features in Kafka. The motivation behind this philosophy is performance and simplicity at the cost of flexibility. In most cases, we can argue that the loss in flexibility is minimal since you can always get that functionality by modeling your application differently, especially if the system supports high performance. ActiveMQ has to support the JMS protocol and hence provide all sorts of hooks and plugins on the brokers at the cost of performance.
Could you elaborate more on your use case? There is probably another way to model your application using Kafka.
Thanks, Neha On Sat, Jun 21, 2014 at 9:24 AM, ravi singh <[EMAIL PROTECTED]> wrote:
Thanks Guozhang/Neha for replies. Here's my use case:
We use proprietary application logging in our apps. We are planning to use Kafka brokers in production , but apart from the logs which are already logged using log4j in kafka we want to log the broker stats using our centralized application logging framework.
Simply put I want to write an application which could start when the kafka brokers starts, read the broker state and metrics and push it to the centralized logging servers.
In ActiveMQ we have a plugin for our proprietary logging. We intercept broker operation and install the plugin into the interceptor chain of the broker.
Regards, Ravi On Mon, Jun 23, 2014 at 9:29 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote: *Regards,* *Ravi*
Primarily we want to log below date(although this is not the exhaustive list):
+ any error/exception during kafka start/stop + any error/exception while broker is running + broker state changes like leader re-election, broker goes down, + Current live brokers + new topic creation + when messages are deleted by broker after specified limit + Broker health : memory usage
Regards, Ravi On Tue, Jun 24, 2014 at 11:11 AM, Neha Narkhede <[EMAIL PROTECTED]> wrote: *Regards,* *Ravi*
Most of these are available via JMX and others can be available via zookeeper. I'm not sure why/how you would monitor "messages being deleted by the broker". In general, monitoring via JMX is preferable to scraping logs.
Thanks, Neha On Mon, Jun 23, 2014 at 11:51 PM, ravi singh <[EMAIL PROTECTED]> wrote: