Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> new log.dirs property (as opposed to log.dir)


Copy link to this message
-
Re: new log.dirs property (as opposed to log.dir)
yeah, that would work!
On Thu, Jun 20, 2013 at 1:20 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Yeah we didn't go as far as adding weighting or anything like that--I
> think we'd be open to a patch that did that as long as it was
> optional. In the short term you can obviously add multiple directories
> on the same disk to increase its share.
>
> -Jay
>
> On Thu, Jun 20, 2013 at 12:59 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> > This sounds like a great idea, to just disks as "just a bunch of disks"
> or
> > JBOD.....hdfs works well this way.
> >
> > Do all the disks need to be the same size, to use them evenly?  Since it
> > will allocate partitions randomly?
> >
> > It would be nice if you had 2 disks, with one twice as large as the
> other,
> > if the larger would be twice as likely to receive partitions as the
> smaller
> > one, etc.
> >
> > I suppose this goes into my earlier question to the list, vis-a-vis
> > heterogeneous brokers (e.g. utilize brokers with different sized storage,
> > using some sort of weighting scheme, etc.).
> >
> > Jason
> >
> >
> > On Thu, Jun 20, 2013 at 11:07 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> >> The intention is to allow the use of multiple disks without RAID or
> >> logical volume management. We have found that there are a lot of
> >> downsides to RAID--in particular a huge throughput hit. Since we
> >> already have a parallelism model due to partitioning and a fault
> >> tolerance model with replication RAID doesn't actually buy much. With
> >> this feature you can directly mount multiple disks as their own
> >> directory and the server will randomly assign partitions to them.
> >>
> >> Obviously this will only work well if there are enough high-throughput
> >> partitions to make load balance evenly (e.g. if you have only one big
> >> partition per server then this isn't going to work).
> >>
> >> -Jay
> >>
> >> On Wed, Jun 19, 2013 at 11:01 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >> wrote:
> >> > is it possible for a partition to have multiple replicas on different
> >> > directories on the same broker?  (hopefully no!)
> >> >
> >> >
> >> > On Wed, Jun 19, 2013 at 10:47 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >> >
> >> >> It takes a comma separated list and partition replicas are randomly
> >> >> distributed to the list.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Jun
> >> >>
> >> >>
> >> >> On Wed, Jun 19, 2013 at 10:25 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >> >> wrote:
> >> >>
> >> >> > In the 0.8 config, log.dir is now log.dirs.  It looks like the
> >> singular
> >> >> > log.dir is still supported, but under the covers the property is
> >> >> log.dirs.
> >> >> >
> >> >> > I'm curious, does this take a comma separated list of directories?
> >>  The
> >> >> new
> >> >> > config page just says:
> >> >> > "The directories in which the log data is kept"
> >> >> >
> >> >> > Also, how does kafka handle multiple directories?  Does it treat
> each
> >> >> > directory as a separate replica partition, or what?
> >> >> >
> >> >> > Jason
> >> >> >
> >> >>
> >>
>