Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> [jira] [Comment Edited] (KAFKA-734) Migration tool needs a revamp, it was poorly written and has many performance bugs


Copy link to this message
-
[jira] [Comment Edited] (KAFKA-734) Migration tool needs a revamp, it was poorly written and has many performance bugs

    [ https://issues.apache.org/jira/browse/KAFKA-734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568900#comment-13568900 ]

Neha Narkhede edited comment on KAFKA-734 at 2/1/13 5:33 PM:
-------------------------------------------------------------

- KafkaMigrationTool

Simplified the partitioning logic of the producers and changed partitionProducers to return a list of producer ids per consumer

- UtilsTest

Added a unit test for the partitioning logic for the producers, that covers all the cases -
producers = consumers
producers < consumers
producers > consumers

- Tested this on a backlog of roughly 100s of GB. Prior to the changes, a migration tool couldn't keep up with production traffic. After applying this patch, it performs well, catches up and can keep up as well
                
      was (Author: nehanarkhede):
    - KafkaMigrationTool

Simplified the partitioning logic of the producers and changed partitionProducers to return a list of producer ids per consumer

- UtilsTest

Added a unit test for the partitioning logic for the producers, that covers all the cases -
producers = consumers
producers < consumers
producers > consumers
                  
> Migration tool needs a revamp, it was poorly written and has many performance bugs
> ----------------------------------------------------------------------------------
>
>                 Key: KAFKA-734
>                 URL: https://issues.apache.org/jira/browse/KAFKA-734
>             Project: Kafka
>          Issue Type: Bug
>          Components: tools
>    Affects Versions: 0.8
>            Reporter: Neha Narkhede
>            Assignee: Neha Narkhede
>            Priority: Blocker
>              Labels: p1
>         Attachments: kafka-734-v1.patch, kafka-734-v2.patch, kafka-734-v3.patch
>
>
> Migration tool has a number of problems ranging from poor logging to poor design. This needs to be thought through again

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB