Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> new log.dirs property (as opposed to log.dir)


Copy link to this message
-
Re: new log.dirs property (as opposed to log.dir)
Yeah we didn't go as far as adding weighting or anything like that--I
think we'd be open to a patch that did that as long as it was
optional. In the short term you can obviously add multiple directories
on the same disk to increase its share.

-Jay

On Thu, Jun 20, 2013 at 12:59 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> This sounds like a great idea, to just disks as "just a bunch of disks" or
> JBOD.....hdfs works well this way.
>
> Do all the disks need to be the same size, to use them evenly?  Since it
> will allocate partitions randomly?
>
> It would be nice if you had 2 disks, with one twice as large as the other,
> if the larger would be twice as likely to receive partitions as the smaller
> one, etc.
>
> I suppose this goes into my earlier question to the list, vis-a-vis
> heterogeneous brokers (e.g. utilize brokers with different sized storage,
> using some sort of weighting scheme, etc.).
>
> Jason
>
>
> On Thu, Jun 20, 2013 at 11:07 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
>> The intention is to allow the use of multiple disks without RAID or
>> logical volume management. We have found that there are a lot of
>> downsides to RAID--in particular a huge throughput hit. Since we
>> already have a parallelism model due to partitioning and a fault
>> tolerance model with replication RAID doesn't actually buy much. With
>> this feature you can directly mount multiple disks as their own
>> directory and the server will randomly assign partitions to them.
>>
>> Obviously this will only work well if there are enough high-throughput
>> partitions to make load balance evenly (e.g. if you have only one big
>> partition per server then this isn't going to work).
>>
>> -Jay
>>
>> On Wed, Jun 19, 2013 at 11:01 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> wrote:
>> > is it possible for a partition to have multiple replicas on different
>> > directories on the same broker?  (hopefully no!)
>> >
>> >
>> > On Wed, Jun 19, 2013 at 10:47 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> >
>> >> It takes a comma separated list and partition replicas are randomly
>> >> distributed to the list.
>> >>
>> >> Thanks,
>> >>
>> >> Jun
>> >>
>> >>
>> >> On Wed, Jun 19, 2013 at 10:25 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> >> wrote:
>> >>
>> >> > In the 0.8 config, log.dir is now log.dirs.  It looks like the
>> singular
>> >> > log.dir is still supported, but under the covers the property is
>> >> log.dirs.
>> >> >
>> >> > I'm curious, does this take a comma separated list of directories?
>>  The
>> >> new
>> >> > config page just says:
>> >> > "The directories in which the log data is kept"
>> >> >
>> >> > Also, how does kafka handle multiple directories?  Does it treat each
>> >> > directory as a separate replica partition, or what?
>> >> >
>> >> > Jason
>> >> >
>> >>
>>