Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> new log.dirs property (as opposed to log.dir)

Copy link to this message
Re: new log.dirs property (as opposed to log.dir)
I believe either should work. The broker has a record of what it should have in zk and will recreate any missing logs. Try it to make sure though.

Sent from my iPhone

On Aug 15, 2013, at 12:52 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Ok, that makes sense that the broker will shut itself down.
> If we bring it back up, can this be with an altered set of log.dirs?  Will
> the destroyed partitions get rebuilt on a new log.dir?  Or do we have to
> bring it back up with a new or repaired disk, matching the old log.dir, in
> order for those replicas to be rebuilt?
> Jason
> On Wed, Aug 14, 2013 at 4:16 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>> If you get a disk error that results in an IOException the broker will
>> shut itself down. You would then have the option of replacing the disk or
>> deleting that data directory from the list. When the broker is brought back
>> up the intact partitions will quickly catch up and be online; the destroyed
>> partitions will have to fully rebuild off the other replicas and will take
>> a little longer but will automatically come back online once they have
>> restored off the replicas.
>> -jay
>> Sent from my iPhone
>> On Aug 14, 2013, at 1:49 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>>> I'm getting ready to try out this configuration (use multiple disks, no
>>> RAID, per broker).  One concern is the procedure for recovering if there
>> is
>>> a disk failure.
>>> If a disk fails, will the broker go offline, or will it continue serving
>>> partitions on its remaining good disks?  And if so, is there a procedure
>>> for moving the partitions that were on the failed disk, but not
>> necessarily
>>> all the others on that broker?
>>> Jason
>>> On Thu, Jun 20, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> wrote:
>>>> yeah, that would work!
>>>> On Thu, Jun 20, 2013 at 1:20 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>>>>> Yeah we didn't go as far as adding weighting or anything like that--I
>>>>> think we'd be open to a patch that did that as long as it was
>>>>> optional. In the short term you can obviously add multiple directories
>>>>> on the same disk to increase its share.
>>>>> -Jay
>>>>> On Thu, Jun 20, 2013 at 12:59 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>> This sounds like a great idea, to just disks as "just a bunch of
>> disks"
>>>>> or
>>>>>> JBOD.....hdfs works well this way.
>>>>>> Do all the disks need to be the same size, to use them evenly?  Since
>> it
>>>>>> will allocate partitions randomly?
>>>>>> It would be nice if you had 2 disks, with one twice as large as the
>>>>> other,
>>>>>> if the larger would be twice as likely to receive partitions as the
>>>>> smaller
>>>>>> one, etc.
>>>>>> I suppose this goes into my earlier question to the list, vis-a-vis
>>>>>> heterogeneous brokers (e.g. utilize brokers with different sized
>>>>> storage,
>>>>>> using some sort of weighting scheme, etc.).
>>>>>> Jason
>>>>>> On Thu, Jun 20, 2013 at 11:07 AM, Jay Kreps <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>>> The intention is to allow the use of multiple disks without RAID or
>>>>>>> logical volume management. We have found that there are a lot of
>>>>>>> downsides to RAID--in particular a huge throughput hit. Since we
>>>>>>> already have a parallelism model due to partitioning and a fault
>>>>>>> tolerance model with replication RAID doesn't actually buy much. With
>>>>>>> this feature you can directly mount multiple disks as their own
>>>>>>> directory and the server will randomly assign partitions to them.
>>>>>>> Obviously this will only work well if there are enough
>> high-throughput
>>>>>>> partitions to make load balance evenly (e.g. if you have only one big
>>>>>>> partition per server then this isn't going to work).
>>>>>>> -Jay
>>>>>>> On Wed, Jun 19, 2013 at 11:01 PM, Jason Rosenberg <[EMAIL PROTECTED]>