Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - new log.dirs property (as opposed to log.dir)


+
Jason Rosenberg 2013-06-20, 05:25
+
Jun Rao 2013-06-20, 05:48
+
Jason Rosenberg 2013-06-20, 06:02
+
Jun Rao 2013-06-20, 14:53
+
Jay Kreps 2013-06-20, 18:08
+
Jason Rosenberg 2013-06-20, 19:59
+
Jay Kreps 2013-06-20, 20:21
+
Jason Rosenberg 2013-06-20, 22:15
+
Jason Rosenberg 2013-08-14, 20:50
+
Jay Kreps 2013-08-14, 23:16
+
Jason Rosenberg 2013-08-15, 07:52
Copy link to this message
-
Re: new log.dirs property (as opposed to log.dir)
Jay Kreps 2013-08-15, 14:11
I believe either should work. The broker has a record of what it should have in zk and will recreate any missing logs. Try it to make sure though.

Sent from my iPhone

On Aug 15, 2013, at 12:52 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Ok, that makes sense that the broker will shut itself down.
>
> If we bring it back up, can this be with an altered set of log.dirs?  Will
> the destroyed partitions get rebuilt on a new log.dir?  Or do we have to
> bring it back up with a new or repaired disk, matching the old log.dir, in
> order for those replicas to be rebuilt?
>
> Jason
>
>
> On Wed, Aug 14, 2013 at 4:16 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
>> If you get a disk error that results in an IOException the broker will
>> shut itself down. You would then have the option of replacing the disk or
>> deleting that data directory from the list. When the broker is brought back
>> up the intact partitions will quickly catch up and be online; the destroyed
>> partitions will have to fully rebuild off the other replicas and will take
>> a little longer but will automatically come back online once they have
>> restored off the replicas.
>>
>> -jay
>>
>> Sent from my iPhone
>>
>> On Aug 14, 2013, at 1:49 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>>
>>> I'm getting ready to try out this configuration (use multiple disks, no
>>> RAID, per broker).  One concern is the procedure for recovering if there
>> is
>>> a disk failure.
>>>
>>> If a disk fails, will the broker go offline, or will it continue serving
>>> partitions on its remaining good disks?  And if so, is there a procedure
>>> for moving the partitions that were on the failed disk, but not
>> necessarily
>>> all the others on that broker?
>>>
>>> Jason
>>>
>>>
>>> On Thu, Jun 20, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> wrote:
>>>
>>>> yeah, that would work!
>>>>
>>>>
>>>> On Thu, Jun 20, 2013 at 1:20 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> Yeah we didn't go as far as adding weighting or anything like that--I
>>>>> think we'd be open to a patch that did that as long as it was
>>>>> optional. In the short term you can obviously add multiple directories
>>>>> on the same disk to increase its share.
>>>>>
>>>>> -Jay
>>>>>
>>>>> On Thu, Jun 20, 2013 at 12:59 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>> This sounds like a great idea, to just disks as "just a bunch of
>> disks"
>>>>> or
>>>>>> JBOD.....hdfs works well this way.
>>>>>>
>>>>>> Do all the disks need to be the same size, to use them evenly?  Since
>> it
>>>>>> will allocate partitions randomly?
>>>>>>
>>>>>> It would be nice if you had 2 disks, with one twice as large as the
>>>>> other,
>>>>>> if the larger would be twice as likely to receive partitions as the
>>>>> smaller
>>>>>> one, etc.
>>>>>>
>>>>>> I suppose this goes into my earlier question to the list, vis-a-vis
>>>>>> heterogeneous brokers (e.g. utilize brokers with different sized
>>>>> storage,
>>>>>> using some sort of weighting scheme, etc.).
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 20, 2013 at 11:07 AM, Jay Kreps <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>>
>>>>>>> The intention is to allow the use of multiple disks without RAID or
>>>>>>> logical volume management. We have found that there are a lot of
>>>>>>> downsides to RAID--in particular a huge throughput hit. Since we
>>>>>>> already have a parallelism model due to partitioning and a fault
>>>>>>> tolerance model with replication RAID doesn't actually buy much. With
>>>>>>> this feature you can directly mount multiple disks as their own
>>>>>>> directory and the server will randomly assign partitions to them.
>>>>>>>
>>>>>>> Obviously this will only work well if there are enough
>> high-throughput
>>>>>>> partitions to make load balance evenly (e.g. if you have only one big
>>>>>>> partition per server then this isn't going to work).
>>>>>>>
>>>>>>> -Jay
>>>>>>>
>>>>>>> On Wed, Jun 19, 2013 at 11:01 PM, Jason Rosenberg <[EMAIL PROTECTED]>

 
+
Jason Rosenberg 2013-08-15, 17:07
+
Vadim Keylis 2013-08-15, 18:29
+
Jay Kreps 2013-08-15, 18:51