Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> slow producing on ec2


Copy link to this message
-
Re: slow producing on ec2
One order of magnitude of difference sounds about right, though, no?

I don't know. I'm by no means a Kafka expert and we don't know much about
the other variables of your situation, so it's hard to tell (at least for
me) if there could be another explanation...

--
Felix

On Tue, Jan 10, 2012 at 5:30 PM, Pierre-Yves Ritschard <[EMAIL PROTECTED]>wrote:

> Surely,
>
> I just didn't expect such dramatically low numbers
>
> On Tue, Jan 10, 2012 at 11:27 PM, Felix GV <[EMAIL PROTECTED]> wrote:
> > Maybe I'm overlooking something, but the first thing that came to my mind
> > is: wouldn't you get no network latency at all on your local box if
> > everything runs on the same machine?
> >
> > On EC2, the network latency would bring your overal throughput down,
> > especially with a sync producer, wouldn't it?
> >
> > --
> > Felix
> >
> >
> >
> > On Tue, Jan 10, 2012 at 5:19 PM, Pierre-Yves Ritschard <[EMAIL PROTECTED]
> >wrote:
> >
> >> Hi,
> >>
> >> One of my cluster runs on ec2 and I noticed that I have rather slow
> >> throughput, less than 500 per second. I isolated the trouble on the
> >> following setup
> >>
> >> 1 producer instance (m1.large)
> >> 5 consumer instances also running a broker (m2.xlarge)
> >> 1 zookeeper instance (m1.large)
> >>
> >> I am using a sync producer and have 60 partitions.
> >>
> >> On my local box, with everything running I would several thousand per
> >> second.
> >>
>