Hi,

Could someone confirm that we dropped the replay of the .oldlogs in the
region directory because it is not needed anymore? Before the master rewrite
we were, just as now, creating the region and for a brief period it had its
own log directory, the .oldlogs under the region directory. The
HRegion.initialize() would replay it in case an edit to the region arrived
in between. It would then also remove it:

  public void initialize(Path initialFiles, final Progressable reporter)
  throws IOException {
    Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
...
    // Play log if one.  Delete when done.
    doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId, reporter);
    if (fs.exists(oldLogFile)) {
      if (LOG.isDebugEnabled()) {
        LOG.debug("Deleting old log file: " + oldLogFile);
      }
      fs.delete(oldLogFile, false);
    }
...

Now we still create that directory, but the above code has been dropped.
Only the recovered.edits is replayed instead while the .oldlogs now hang
around. I am not even sure when they may contain an entry, but if they don't
then we can either do not use "<regiondir>/.oldlogs", but point it to the
new "<regiondir>/.tmp" - or we can add a unconditional delete in the
initialize as before.

Create a JIRA for this?

Lars

Hi,

Ah, I see Ted has that also questioned in HBASE-4010. Good.

And I was slightly wrong below, as there is a .logs and .oldlogs for the newly created regions. I agree that when there is no chance for the region to take edits before it gets the proper log, then we should drop this. Those dangling dirs do not harm, but also do not help either.

Lars

On Jul 5, 2011, at 8:39 AM, Lars George wrote:

> Hi,
>
> Could someone confirm that we dropped the replay of the .oldlogs in the region directory because it is not needed anymore? Before the master rewrite we were, just as now, creating the region and for a brief period it had its own log directory, the .oldlogs under the region directory. The HRegion.initialize() would replay it in case an edit to the region arrived in between. It would then also remove it:
>
>   public void initialize(Path initialFiles, final Progressable reporter)
>   throws IOException {
>     Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
> ...
>     // Play log if one.  Delete when done.
>     doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId, reporter);
>     if (fs.exists(oldLogFile)) {
>       if (LOG.isDebugEnabled()) {
>         LOG.debug("Deleting old log file: " + oldLogFile);
>       }
>       fs.delete(oldLogFile, false);
>     }
> ...
>
> Now we still create that directory, but the above code has been dropped. Only the recovered.edits is replayed instead while the .oldlogs now hang around. I am not even sure when they may contain an entry, but if they don't then we can either do not use "<regiondir>/.oldlogs", but point it to the new "<regiondir>/.tmp" - or we can add a unconditional delete in the initialize as before.
>
> Create a JIRA for this?
>
> Lars

Doesn't the wide-area HBase replication use the .oldlogs to keep the slave
hBase cluster in sync?

thanks
dhruba

On Tue, Jul 5, 2011 at 7:27 AM, Lars George <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Ah, I see Ted has that also questioned in HBASE-4010. Good.
>
> And I was slightly wrong below, as there is a .logs and .oldlogs for the
> newly created regions. I agree that when there is no chance for the region
> to take edits before it gets the proper log, then we should drop this. Those
> dangling dirs do not harm, but also do not help either.
>
> Lars
>
> On Jul 5, 2011, at 8:39 AM, Lars George wrote:
>
> > Hi,
> >
> > Could someone confirm that we dropped the replay of the .oldlogs in the
> region directory because it is not needed anymore? Before the master rewrite
> we were, just as now, creating the region and for a brief period it had its
> own log directory, the .oldlogs under the region directory. The
> HRegion.initialize() would replay it in case an edit to the region arrived
> in between. It would then also remove it:
> >
> >   public void initialize(Path initialFiles, final Progressable reporter)
> >   throws IOException {
> >     Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
> > ...
> >     // Play log if one.  Delete when done.
> >     doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId,
> reporter);
> >     if (fs.exists(oldLogFile)) {
> >       if (LOG.isDebugEnabled()) {
> >         LOG.debug("Deleting old log file: " + oldLogFile);
> >       }
> >       fs.delete(oldLogFile, false);
> >     }
> > ...
> >
> > Now we still create that directory, but the above code has been dropped.
> Only the recovered.edits is replayed instead while the .oldlogs now hang
> around. I am not even sure when they may contain an entry, but if they don't
> then we can either do not use "<regiondir>/.oldlogs", but point it to the
> new "<regiondir>/.tmp" - or we can add a unconditional delete in the
> initialize as before.
> >
> > Create a JIRA for this?
> >
> > Lars
>
>
--
Connect to me at http://www.facebook.com/dhruba

Hi Dhruba,

Yes, but that is the one on the root dir level. I am referring to the one on
the region level.

Cheers,
Lars

On Tue, Jul 5, 2011 at 6:25 AM, Dhruba Borthakur <[EMAIL PROTECTED]> wrote:

> Doesn't the wide-area HBase replication use the .oldlogs to keep the slave
> hBase cluster in sync?
>
> thanks
> dhruba
>
> On Tue, Jul 5, 2011 at 7:27 AM, Lars George <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > Ah, I see Ted has that also questioned in HBASE-4010. Good.
> >
> > And I was slightly wrong below, as there is a .logs and .oldlogs for the
> > newly created regions. I agree that when there is no chance for the
> region
> > to take edits before it gets the proper log, then we should drop this.
> Those
> > dangling dirs do not harm, but also do not help either.
> >
> > Lars
> >
> > On Jul 5, 2011, at 8:39 AM, Lars George wrote:
> >
> > > Hi,
> > >
> > > Could someone confirm that we dropped the replay of the .oldlogs in the
> > region directory because it is not needed anymore? Before the master
> rewrite
> > we were, just as now, creating the region and for a brief period it had
> its
> > own log directory, the .oldlogs under the region directory. The
> > HRegion.initialize() would replay it in case an edit to the region
> arrived
> > in between. It would then also remove it:
> > >
> > >   public void initialize(Path initialFiles, final Progressable
> reporter)
> > >   throws IOException {
> > >     Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
> > > ...
> > >     // Play log if one.  Delete when done.
> > >     doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId,
> > reporter);
> > >     if (fs.exists(oldLogFile)) {
> > >       if (LOG.isDebugEnabled()) {
> > >         LOG.debug("Deleting old log file: " + oldLogFile);
> > >       }
> > >       fs.delete(oldLogFile, false);
> > >     }
> > > ...
> > >
> > > Now we still create that directory, but the above code has been
> dropped.
> > Only the recovered.edits is replayed instead while the .oldlogs now hang
> > around. I am not even sure when they may contain an entry, but if they
> don't
> > then we can either do not use "<regiondir>/.oldlogs", but point it to the
> > new "<regiondir>/.tmp" - or we can add a unconditional delete in the
> > initialize as before.
> > >
> > > Create a JIRA for this?
> > >
> > > Lars
> >
> >
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

Ok, got it, thanks Lars.

-dhruba

On Tue, Jul 5, 2011 at 2:41 PM, Lars George <[EMAIL PROTECTED]> wrote:

> Hi Dhruba,
>
> Yes, but that is the one on the root dir level. I am referring to the one
> on
> the region level.
>
> Cheers,
> Lars
>
> On Tue, Jul 5, 2011 at 6:25 AM, Dhruba Borthakur <[EMAIL PROTECTED]> wrote:
>
> > Doesn't the wide-area HBase replication use the .oldlogs to keep the
> slave
> > hBase cluster in sync?
> >
> > thanks
> > dhruba
> >
> > On Tue, Jul 5, 2011 at 7:27 AM, Lars George <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hi,
> > >
> > > Ah, I see Ted has that also questioned in HBASE-4010. Good.
> > >
> > > And I was slightly wrong below, as there is a .logs and .oldlogs for
> the
> > > newly created regions. I agree that when there is no chance for the
> > region
> > > to take edits before it gets the proper log, then we should drop this.
> > Those
> > > dangling dirs do not harm, but also do not help either.
> > >
> > > Lars
> > >
> > > On Jul 5, 2011, at 8:39 AM, Lars George wrote:
> > >
> > > > Hi,
> > > >
> > > > Could someone confirm that we dropped the replay of the .oldlogs in
> the
> > > region directory because it is not needed anymore? Before the master
> > rewrite
> > > we were, just as now, creating the region and for a brief period it had
> > its
> > > own log directory, the .oldlogs under the region directory. The
> > > HRegion.initialize() would replay it in case an edit to the region
> > arrived
> > > in between. It would then also remove it:
> > > >
> > > >   public void initialize(Path initialFiles, final Progressable
> > reporter)
> > > >   throws IOException {
> > > >     Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
> > > > ...
> > > >     // Play log if one.  Delete when done.
> > > >     doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId,
> > > reporter);
> > > >     if (fs.exists(oldLogFile)) {
> > > >       if (LOG.isDebugEnabled()) {
> > > >         LOG.debug("Deleting old log file: " + oldLogFile);
> > > >       }
> > > >       fs.delete(oldLogFile, false);
> > > >     }
> > > > ...
> > > >
> > > > Now we still create that directory, but the above code has been
> > dropped.
> > > Only the recovered.edits is replayed instead while the .oldlogs now
> hang
> > > around. I am not even sure when they may contain an entry, but if they
> > don't
> > > then we can either do not use "<regiondir>/.oldlogs", but point it to
> the
> > > new "<regiondir>/.tmp" - or we can add a unconditional delete in the
> > > initialize as before.
> > > >
> > > > Create a JIRA for this?
> > > >
> > > > Lars
> > >
> > >
> >
> >
> > --
> > Connect to me at http://www.facebook.com/dhruba
> >
>

--
Connect to me at http://www.facebook.com/dhruba

It looks like left-over crud Lars.  Thanks for noting it.  I made
HBASE-4068 to clean it up.
St.Ack

On Tue, Jul 5, 2011 at 7:41 AM, Lars George <[EMAIL PROTECTED]> wrote:
> Hi Dhruba,
>
> Yes, but that is the one on the root dir level. I am referring to the one on
> the region level.
>
> Cheers,
> Lars
>
> On Tue, Jul 5, 2011 at 6:25 AM, Dhruba Borthakur <[EMAIL PROTECTED]> wrote:
>
>> Doesn't the wide-area HBase replication use the .oldlogs to keep the slave
>> hBase cluster in sync?
>>
>> thanks
>> dhruba
>>
>> On Tue, Jul 5, 2011 at 7:27 AM, Lars George <[EMAIL PROTECTED]> wrote:
>>
>> > Hi,
>> >
>> > Ah, I see Ted has that also questioned in HBASE-4010. Good.
>> >
>> > And I was slightly wrong below, as there is a .logs and .oldlogs for the
>> > newly created regions. I agree that when there is no chance for the
>> region
>> > to take edits before it gets the proper log, then we should drop this.
>> Those
>> > dangling dirs do not harm, but also do not help either.
>> >
>> > Lars
>> >
>> > On Jul 5, 2011, at 8:39 AM, Lars George wrote:
>> >
>> > > Hi,
>> > >
>> > > Could someone confirm that we dropped the replay of the .oldlogs in the
>> > region directory because it is not needed anymore? Before the master
>> rewrite
>> > we were, just as now, creating the region and for a brief period it had
>> its
>> > own log directory, the .oldlogs under the region directory. The
>> > HRegion.initialize() would replay it in case an edit to the region
>> arrived
>> > in between. It would then also remove it:
>> > >
>> > >   public void initialize(Path initialFiles, final Progressable
>> reporter)
>> > >   throws IOException {
>> > >     Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
>> > > ...
>> > >     // Play log if one.  Delete when done.
>> > >     doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId,
>> > reporter);
>> > >     if (fs.exists(oldLogFile)) {
>> > >       if (LOG.isDebugEnabled()) {
>> > >         LOG.debug("Deleting old log file: " + oldLogFile);
>> > >       }
>> > >       fs.delete(oldLogFile, false);
>> > >     }
>> > > ...
>> > >
>> > > Now we still create that directory, but the above code has been
>> dropped.
>> > Only the recovered.edits is replayed instead while the .oldlogs now hang
>> > around. I am not even sure when they may contain an entry, but if they
>> don't
>> > then we can either do not use "<regiondir>/.oldlogs", but point it to the
>> > new "<regiondir>/.tmp" - or we can add a unconditional delete in the
>> > initialize as before.
>> > >
>> > > Create a JIRA for this?
>> > >
>> > > Lars
>> >
>> >
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>