Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Performance of jBOD


Copy link to this message
-
Re: Performance of jBOD
The average observed per broker write bandwidth is roughly 5 MB/s and read
bandwidth is roughly 20 MB/s. But
these statistics are from clusters that are no where close to maxing out
the write/read bandwidth available. In most cases though, you will hit the
network bottleneck (~125 MB/s) first.

Thanks,
Neha
On Thu, Oct 3, 2013 at 11:08 AM, Graeme Wallace <
[EMAIL PROTECTED]> wrote:

> Jun,
>
> Just to follow up - have you any stats on what disk read + write throughput
> you are getting in production at LinkedIn ?
>
> regards,
>
> Graeme
>
>
>
> On Thu, Oct 3, 2013 at 7:17 AM, Graeme Wallace <
> [EMAIL PROTECTED]> wrote:
>
> > Its whatever dstat reports - i think it must be MBytes ?
> >
> >
> > On Wed, Oct 2, 2013 at 10:29 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> If you use a reasonable fetch size in the consumer (e.g. 100+KB) or
> above,
> >> you could probably get most of the sequential scan performance from
> those
> >> disks. Do you mean 250 Mbits or MBytes?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Wed, Oct 2, 2013 at 4:35 PM, Graeme Wallace <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >> > We notice that we get a lot higher performance from Kafka up until the
> >> > point where disk reads are taking place. (which is to be expected as
> its
> >> > reading from filesystem cache).
> >> >
> >> > However, given that we have 12x7200rpm SATA disks configured as jBOD -
> >> what
> >> > sort of disk read performance should i see reported by dstat when
> things
> >> > are running flat out ?
> >> >
> >> > I'm seeing 250-340Mb sustained.
> >> >
> >> > --
> >> > Graeme Wallace
> >> > CTO
> >> > FareCompare.com
> >> > O: 972 588 1414
> >> > M: 214 681 9018
> >> >
> >>
> >
> >
> >
> > --
> > Graeme Wallace
> > CTO
> > FareCompare.com
> > O: 972 588 1414
> > M: 214 681 9018
> >
> >
>
>
> --
> Graeme Wallace
> CTO
> FareCompare.com
> O: 972 588 1414
> M: 214 681 9018
>