Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> [jira] [Updated] (KAFKA-683) Fix correlation ids in all requests sent to kafka


Copy link to this message
-
[jira] [Updated] (KAFKA-683) Fix correlation ids in all requests sent to kafka

     [ https://issues.apache.org/jira/browse/KAFKA-683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neha Narkhede updated KAFKA-683:
--------------------------------

    Attachment: kafka-683-v1.patch

This patch does the following -
1. Fixes the correlation id for all requests. The correlation id is not supposed to be globally unique, just unique per client for a sufficiently long time. Hence, for the controller, it is initialized to zero upon startup. For fetch requests, it is initialized on the builder creation. For produce requests, it is initialized on producer object creation.
2. Fixes logging to output a kafka request log and a state change log. Kafka request log is a daily rolling file appender that contains every request received and served by a kafka broker. State change log contains log statements coming from the controller so that it doesn't pollute the server log.
3. Fixed a bug in StopReplicaRequest since it didn't write the correlation id as part of serializing the request
4. Added some log4j statements to kafka.log to specify the reason for rolling a log segment and to say when we create and delete data and index files
5. Fixed the client id for the fetchers inside ZookeeperConsumerConnector to have not just the wired in client id but also the fetcher thread name. So the requests sent by the individual fetchers from the consumer will appear with the read name in the Kafka broker's request log. This will allow us to correlate threads from thread dumps with the requests it sent to the broker. This is helpful during debugging.
                
> Fix correlation ids in all requests sent to kafka
> -------------------------------------------------
>
>                 Key: KAFKA-683
>                 URL: https://issues.apache.org/jira/browse/KAFKA-683
>             Project: Kafka
>          Issue Type: Improvement
>    Affects Versions: 0.8
>            Reporter: Neha Narkhede
>            Assignee: Neha Narkhede
>            Priority: Critical
>              Labels: improvement, replication
>         Attachments: kafka-683-v1.patch
>
>
> We should fix the correlation ids in every request sent to Kafka and fix the request log on the broker to specify not only the type of request and who sent it, but also the correlation id. This will be very helpful while troubleshooting problems in production.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB