Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Mirrormaker 0.7


+
Walter King 2013-08-30, 00:09
Copy link to this message
-
Re: Mirrormaker 0.7
Jun Rao 2013-08-30, 03:55
The following may help:

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Howtoimprovethethroughputofaremoteconsumer%3F

Thanks,

Jun
On Thu, Aug 29, 2013 at 4:47 PM, Walter King <[EMAIL PROTECTED]> wrote:

> I'm having a lot of trouble getting throughput of mirrormaker to be
> consistent.  On ec2, I have ~10 topics with pretty different throughputs,
> mirroring from 4 regions into 1.  Total of about 20MBps. Sometimes it seems
> to be going fine, and then it seems like one of the consumers stops
> altogether, and restarting the process will let it catch back up.
>  Sometimes I'll see errors in the logs about mirrormaker threads being
> stopped, and it doesn't seem like they ever get replaced.
>
> Whats the recommended setup for mirrormaker? I've tried changing number of
> streams, producers, adding servers, splitting processes by topic or region,
>  all to varying degrees of effect but still seem to just lose a partition
> periodically.
>
> Thanks
>