Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Mirrormaker 0.7


Copy link to this message
-
Re: Mirrormaker 0.7
The following may help:

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Howtoimprovethethroughputofaremoteconsumer%3F

Thanks,

Jun
On Thu, Aug 29, 2013 at 4:47 PM, Walter King <[EMAIL PROTECTED]> wrote:

> I'm having a lot of trouble getting throughput of mirrormaker to be
> consistent.  On ec2, I have ~10 topics with pretty different throughputs,
> mirroring from 4 regions into 1.  Total of about 20MBps. Sometimes it seems
> to be going fine, and then it seems like one of the consumers stops
> altogether, and restarting the process will let it catch back up.
>  Sometimes I'll see errors in the logs about mirrormaker threads being
> stopped, and it doesn't seem like they ever get replaced.
>
> Whats the recommended setup for mirrormaker? I've tried changing number of
> streams, producers, adding servers, splitting processes by topic or region,
>  all to varying degrees of effect but still seem to just lose a partition
> periodically.
>
> Thanks
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB