Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> new log.dirs property (as opposed to log.dir)


Copy link to this message
-
Re: new log.dirs property (as opposed to log.dir)
Thanks Jay, I'll do some testing with this and report back.

Jason
On Thu, Aug 15, 2013 at 7:10 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> I believe either should work. The broker has a record of what it should
> have in zk and will recreate any missing logs. Try it to make sure though.
>
> Sent from my iPhone
>
> On Aug 15, 2013, at 12:52 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Ok, that makes sense that the broker will shut itself down.
> >
> > If we bring it back up, can this be with an altered set of log.dirs?
>  Will
> > the destroyed partitions get rebuilt on a new log.dir?  Or do we have to
> > bring it back up with a new or repaired disk, matching the old log.dir,
> in
> > order for those replicas to be rebuilt?
> >
> > Jason
> >
> >
> > On Wed, Aug 14, 2013 at 4:16 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> >> If you get a disk error that results in an IOException the broker will
> >> shut itself down. You would then have the option of replacing the disk
> or
> >> deleting that data directory from the list. When the broker is brought
> back
> >> up the intact partitions will quickly catch up and be online; the
> destroyed
> >> partitions will have to fully rebuild off the other replicas and will
> take
> >> a little longer but will automatically come back online once they have
> >> restored off the replicas.
> >>
> >> -jay
> >>
> >> Sent from my iPhone
> >>
> >> On Aug 14, 2013, at 1:49 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> >>
> >>> I'm getting ready to try out this configuration (use multiple disks, no
> >>> RAID, per broker).  One concern is the procedure for recovering if
> there
> >> is
> >>> a disk failure.
> >>>
> >>> If a disk fails, will the broker go offline, or will it continue
> serving
> >>> partitions on its remaining good disks?  And if so, is there a
> procedure
> >>> for moving the partitions that were on the failed disk, but not
> >> necessarily
> >>> all the others on that broker?
> >>>
> >>> Jason
> >>>
> >>>
> >>> On Thu, Jun 20, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >> wrote:
> >>>
> >>>> yeah, that would work!
> >>>>
> >>>>
> >>>> On Thu, Jun 20, 2013 at 1:20 PM, Jay Kreps <[EMAIL PROTECTED]>
> wrote:
> >>>>
> >>>>> Yeah we didn't go as far as adding weighting or anything like that--I
> >>>>> think we'd be open to a patch that did that as long as it was
> >>>>> optional. In the short term you can obviously add multiple
> directories
> >>>>> on the same disk to increase its share.
> >>>>>
> >>>>> -Jay
> >>>>>
> >>>>> On Thu, Jun 20, 2013 at 12:59 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >>>>> wrote:
> >>>>>> This sounds like a great idea, to just disks as "just a bunch of
> >> disks"
> >>>>> or
> >>>>>> JBOD.....hdfs works well this way.
> >>>>>>
> >>>>>> Do all the disks need to be the same size, to use them evenly?
>  Since
> >> it
> >>>>>> will allocate partitions randomly?
> >>>>>>
> >>>>>> It would be nice if you had 2 disks, with one twice as large as the
> >>>>> other,
> >>>>>> if the larger would be twice as likely to receive partitions as the
> >>>>> smaller
> >>>>>> one, etc.
> >>>>>>
> >>>>>> I suppose this goes into my earlier question to the list, vis-a-vis
> >>>>>> heterogeneous brokers (e.g. utilize brokers with different sized
> >>>>> storage,
> >>>>>> using some sort of weighting scheme, etc.).
> >>>>>>
> >>>>>> Jason
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Jun 20, 2013 at 11:07 AM, Jay Kreps <[EMAIL PROTECTED]>
> >>>>> wrote:
> >>>>>>
> >>>>>>> The intention is to allow the use of multiple disks without RAID or
> >>>>>>> logical volume management. We have found that there are a lot of
> >>>>>>> downsides to RAID--in particular a huge throughput hit. Since we
> >>>>>>> already have a parallelism model due to partitioning and a fault
> >>>>>>> tolerance model with replication RAID doesn't actually buy much.
> With
> >>>>>>> this feature you can directly mount multiple disks as their own
> >>>>>>> directory and the server will randomly assign partitions to them.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB