Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - new log.dirs property (as opposed to log.dir)


Copy link to this message
-
Re: new log.dirs property (as opposed to log.dir)
Jason Rosenberg 2013-08-15, 17:07
Thanks Jay, I'll do some testing with this and report back.

Jason
On Thu, Aug 15, 2013 at 7:10 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> I believe either should work. The broker has a record of what it should
> have in zk and will recreate any missing logs. Try it to make sure though.
>
> Sent from my iPhone
>
> On Aug 15, 2013, at 12:52 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Ok, that makes sense that the broker will shut itself down.
> >
> > If we bring it back up, can this be with an altered set of log.dirs?
>  Will
> > the destroyed partitions get rebuilt on a new log.dir?  Or do we have to
> > bring it back up with a new or repaired disk, matching the old log.dir,
> in
> > order for those replicas to be rebuilt?
> >
> > Jason
> >
> >
> > On Wed, Aug 14, 2013 at 4:16 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> >> If you get a disk error that results in an IOException the broker will
> >> shut itself down. You would then have the option of replacing the disk
> or
> >> deleting that data directory from the list. When the broker is brought
> back
> >> up the intact partitions will quickly catch up and be online; the
> destroyed
> >> partitions will have to fully rebuild off the other replicas and will
> take
> >> a little longer but will automatically come back online once they have
> >> restored off the replicas.
> >>
> >> -jay
> >>
> >> Sent from my iPhone
> >>
> >> On Aug 14, 2013, at 1:49 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> >>
> >>> I'm getting ready to try out this configuration (use multiple disks, no
> >>> RAID, per broker).  One concern is the procedure for recovering if
> there
> >> is
> >>> a disk failure.
> >>>
> >>> If a disk fails, will the broker go offline, or will it continue
> serving
> >>> partitions on its remaining good disks?  And if so, is there a
> procedure
> >>> for moving the partitions that were on the failed disk, but not
> >> necessarily
> >>> all the others on that broker?
> >>>
> >>> Jason
> >>>
> >>>
> >>> On Thu, Jun 20, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >> wrote:
> >>>
> >>>> yeah, that would work!
> >>>>
> >>>>
> >>>> On Thu, Jun 20, 2013 at 1:20 PM, Jay Kreps <[EMAIL PROTECTED]>
> wrote:
> >>>>
> >>>>> Yeah we didn't go as far as adding weighting or anything like that--I
> >>>>> think we'd be open to a patch that did that as long as it was
> >>>>> optional. In the short term you can obviously add multiple
> directories
> >>>>> on the same disk to increase its share.
> >>>>>
> >>>>> -Jay
> >>>>>
> >>>>> On Thu, Jun 20, 2013 at 12:59 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >>>>> wrote:
> >>>>>> This sounds like a great idea, to just disks as "just a bunch of
> >> disks"
> >>>>> or
> >>>>>> JBOD.....hdfs works well this way.
> >>>>>>
> >>>>>> Do all the disks need to be the same size, to use them evenly?
>  Since
> >> it
> >>>>>> will allocate partitions randomly?
> >>>>>>
> >>>>>> It would be nice if you had 2 disks, with one twice as large as the
> >>>>> other,
> >>>>>> if the larger would be twice as likely to receive partitions as the
> >>>>> smaller
> >>>>>> one, etc.
> >>>>>>
> >>>>>> I suppose this goes into my earlier question to the list, vis-a-vis
> >>>>>> heterogeneous brokers (e.g. utilize brokers with different sized
> >>>>> storage,
> >>>>>> using some sort of weighting scheme, etc.).
> >>>>>>
> >>>>>> Jason
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Jun 20, 2013 at 11:07 AM, Jay Kreps <[EMAIL PROTECTED]>
> >>>>> wrote:
> >>>>>>
> >>>>>>> The intention is to allow the use of multiple disks without RAID or
> >>>>>>> logical volume management. We have found that there are a lot of
> >>>>>>> downsides to RAID--in particular a huge throughput hit. Since we
> >>>>>>> already have a parallelism model due to partitioning and a fault
> >>>>>>> tolerance model with replication RAID doesn't actually buy much.
> With
> >>>>>>> this feature you can directly mount multiple disks as their own
> >>>>>>> directory and the server will randomly assign partitions to them.