Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Mirrormaker 0.7

Walter King 2013-08-30, 00:09
Copy link to this message
Re: Mirrormaker 0.7
The following may help:



On Thu, Aug 29, 2013 at 4:47 PM, Walter King <[EMAIL PROTECTED]> wrote:

> I'm having a lot of trouble getting throughput of mirrormaker to be
> consistent.  On ec2, I have ~10 topics with pretty different throughputs,
> mirroring from 4 regions into 1.  Total of about 20MBps. Sometimes it seems
> to be going fine, and then it seems like one of the consumers stops
> altogether, and restarting the process will let it catch back up.
>  Sometimes I'll see errors in the logs about mirrormaker threads being
> stopped, and it doesn't seem like they ever get replaced.
> Whats the recommended setup for mirrormaker? I've tried changing number of
> streams, producers, adding servers, splitting processes by topic or region,
>  all to varying degrees of effect but still seem to just lose a partition
> periodically.
> Thanks