Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Performance of jBOD


Copy link to this message
-
Re: Performance of jBOD
The average observed per broker write bandwidth is roughly 5 MB/s and read
bandwidth is roughly 20 MB/s. But
these statistics are from clusters that are no where close to maxing out
the write/read bandwidth available. In most cases though, you will hit the
network bottleneck (~125 MB/s) first.

Thanks,
Neha
On Thu, Oct 3, 2013 at 11:08 AM, Graeme Wallace <
[EMAIL PROTECTED]> wrote:

> Jun,
>
> Just to follow up - have you any stats on what disk read + write throughput
> you are getting in production at LinkedIn ?
>
> regards,
>
> Graeme
>
>
>
> On Thu, Oct 3, 2013 at 7:17 AM, Graeme Wallace <
> [EMAIL PROTECTED]> wrote:
>
> > Its whatever dstat reports - i think it must be MBytes ?
> >
> >
> > On Wed, Oct 2, 2013 at 10:29 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> If you use a reasonable fetch size in the consumer (e.g. 100+KB) or
> above,
> >> you could probably get most of the sequential scan performance from
> those
> >> disks. Do you mean 250 Mbits or MBytes?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Wed, Oct 2, 2013 at 4:35 PM, Graeme Wallace <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >> > We notice that we get a lot higher performance from Kafka up until the
> >> > point where disk reads are taking place. (which is to be expected as
> its
> >> > reading from filesystem cache).
> >> >
> >> > However, given that we have 12x7200rpm SATA disks configured as jBOD -
> >> what
> >> > sort of disk read performance should i see reported by dstat when
> things
> >> > are running flat out ?
> >> >
> >> > I'm seeing 250-340Mb sustained.
> >> >
> >> > --
> >> > Graeme Wallace
> >> > CTO
> >> > FareCompare.com
> >> > O: 972 588 1414
> >> > M: 214 681 9018
> >> >
> >>
> >
> >
> >
> > --
> > Graeme Wallace
> > CTO
> > FareCompare.com
> > O: 972 588 1414
> > M: 214 681 9018
> >
> >
>
>
> --
> Graeme Wallace
> CTO
> FareCompare.com
> O: 972 588 1414
> M: 214 681 9018
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB