Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> new log.dirs property (as opposed to log.dir)


+
Jason Rosenberg 2013-06-20, 05:25
+
Jun Rao 2013-06-20, 05:48
+
Jason Rosenberg 2013-06-20, 06:02
+
Jun Rao 2013-06-20, 14:53
+
Jay Kreps 2013-06-20, 18:08
+
Jason Rosenberg 2013-06-20, 19:59
+
Jay Kreps 2013-06-20, 20:21
Copy link to this message
-
Re: new log.dirs property (as opposed to log.dir)
yeah, that would work!
On Thu, Jun 20, 2013 at 1:20 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Yeah we didn't go as far as adding weighting or anything like that--I
> think we'd be open to a patch that did that as long as it was
> optional. In the short term you can obviously add multiple directories
> on the same disk to increase its share.
>
> -Jay
>
> On Thu, Jun 20, 2013 at 12:59 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> > This sounds like a great idea, to just disks as "just a bunch of disks"
> or
> > JBOD.....hdfs works well this way.
> >
> > Do all the disks need to be the same size, to use them evenly?  Since it
> > will allocate partitions randomly?
> >
> > It would be nice if you had 2 disks, with one twice as large as the
> other,
> > if the larger would be twice as likely to receive partitions as the
> smaller
> > one, etc.
> >
> > I suppose this goes into my earlier question to the list, vis-a-vis
> > heterogeneous brokers (e.g. utilize brokers with different sized storage,
> > using some sort of weighting scheme, etc.).
> >
> > Jason
> >
> >
> > On Thu, Jun 20, 2013 at 11:07 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> >> The intention is to allow the use of multiple disks without RAID or
> >> logical volume management. We have found that there are a lot of
> >> downsides to RAID--in particular a huge throughput hit. Since we
> >> already have a parallelism model due to partitioning and a fault
> >> tolerance model with replication RAID doesn't actually buy much. With
> >> this feature you can directly mount multiple disks as their own
> >> directory and the server will randomly assign partitions to them.
> >>
> >> Obviously this will only work well if there are enough high-throughput
> >> partitions to make load balance evenly (e.g. if you have only one big
> >> partition per server then this isn't going to work).
> >>
> >> -Jay
> >>
> >> On Wed, Jun 19, 2013 at 11:01 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >> wrote:
> >> > is it possible for a partition to have multiple replicas on different
> >> > directories on the same broker?  (hopefully no!)
> >> >
> >> >
> >> > On Wed, Jun 19, 2013 at 10:47 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >> >
> >> >> It takes a comma separated list and partition replicas are randomly
> >> >> distributed to the list.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Jun
> >> >>
> >> >>
> >> >> On Wed, Jun 19, 2013 at 10:25 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >> >> wrote:
> >> >>
> >> >> > In the 0.8 config, log.dir is now log.dirs.  It looks like the
> >> singular
> >> >> > log.dir is still supported, but under the covers the property is
> >> >> log.dirs.
> >> >> >
> >> >> > I'm curious, does this take a comma separated list of directories?
> >>  The
> >> >> new
> >> >> > config page just says:
> >> >> > "The directories in which the log data is kept"
> >> >> >
> >> >> > Also, how does kafka handle multiple directories?  Does it treat
> each
> >> >> > directory as a separate replica partition, or what?
> >> >> >
> >> >> > Jason
> >> >> >
> >> >>
> >>
>

 
+
Jason Rosenberg 2013-08-14, 20:50
+
Jay Kreps 2013-08-14, 23:16
+
Jason Rosenberg 2013-08-15, 07:52
+
Jay Kreps 2013-08-15, 14:11
+
Jason Rosenberg 2013-08-15, 17:07
+
Vadim Keylis 2013-08-15, 18:29
+
Jay Kreps 2013-08-15, 18:51
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB