I am changing the subject to reflect the discussion...

If we only load data in bulk (that is, via doBulkLoad(), not using
TableOutputFormat), do we still risk data loss? My understanding is that
append is needed for the WAL, and the WAL is needed only for puts. But bulk
loads bypass the WAL.

For instance, when a region is split, the master must write the new meta
data to the meta regions. Would that require a WAL or rely on append in some
other way?

Are there other situations where the WAL is needed (or append is needed) to
avoid data loss?

Thanks -Andreas.
On Tue, Jun 21, 2011 at 2:29 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> > From: Francis Christopher Liu <[EMAIL PROTECTED]>
> > Thanks for the warning, we'd like to stick with the ASF
> > releases of hadoop.
>
> That's not really advisable with HBase. It's a touchy subject, the 0.20-ish
> support for append in HDFS exists in production at some large places but
> isn't in any ASF 0.20.
>
> We have replayed the append branch on top of branch-0.20-security:
>  https://github.com/trendmicro/hadoop-common/tree/0.20-security-append
>
> This is a companion ASF-ish branch to our in development branch of HBase
> for HBASE-3025 (security and access control):
>  http://github.com/trendmicro/hbase/tree/security
>
> If you're willing to risk data loss with HBase anyway you should give this
> a shot. We really need a branch-0.20-security-append.
>
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
>

> From: Andreas Neumann <[EMAIL PROTECTED]>
> If we only load data in bulk (that is, via doBulkLoad(), not using
> TableOutputFormat), do we still risk data loss? My understanding is
> that append is needed for the WAL, and the WAL is needed only
> for puts. But bulk loads bypass the WAL.

Correct.

If you are doing read-only serving of HFiles built by MR and loaded by doBulkLoad, then you would not need append support.

If adding new data to tables via the HBase API, then sooner or later this will change table structure, which is recorded via Puts to META, which is self-hosted. Circumstances where those edits can be lost without working append support in HDFS may be rare but not rare enough in my estimation. Losing edits to META is bad. This can lead to missing regions and hung clients. Human intervention will be necessary and the time scale for administrative recovery is usually an availability problem.

> For instance, when a region is split, the master must write
> the new meta data to the meta regions. Would that require a WAL
> or rely on append in some other way?

See above.

> Are there other situations where the WAL is needed (or append
> is needed) to avoid data loss?

Deletes? Increments? For these operations you would not lose data per se if you don't have append support, but the client may be incorrectly led to believe they were successfully applied under the same low probability failure conditions that can corrupt META.

  - Andy

Thanks Andy for the clear response.

We are indeed going to use bulk load only, and no puts, deletes or
increments. So the only puts we will have are those that are caused by
changes in the table structure. I guess that includes region splits but also
reassignment of a region after its region server died.

I agree that even though these are rare, they are not rare enough to take a
risk. But they could be rare enough to justify a less efficient
implementation of the WAL. Would it be reasonable to use an implementation
of HLog that - at the price of performance - persists the WAL to HDFS
without relying on append?

Cheers -Andreas.
On Wed, Jun 22, 2011 at 4:36 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> > From: Andreas Neumann <[EMAIL PROTECTED]>
> > If we only load data in bulk (that is, via doBulkLoad(), not using
> > TableOutputFormat), do we still risk data loss? My understanding is
> > that append is needed for the WAL, and the WAL is needed only
> > for puts. But bulk loads bypass the WAL.
>
> Correct.
>
> If you are doing read-only serving of HFiles built by MR and loaded by
> doBulkLoad, then you would not need append support.
>
> If adding new data to tables via the HBase API, then sooner or later this
> will change table structure, which is recorded via Puts to META, which is
> self-hosted. Circumstances where those edits can be lost without working
> append support in HDFS may be rare but not rare enough in my estimation.
> Losing edits to META is bad. This can lead to missing regions and hung
> clients. Human intervention will be necessary and the time scale for
> administrative recovery is usually an availability problem.
>
> > For instance, when a region is split, the master must write
> > the new meta data to the meta regions. Would that require a WAL
> > or rely on append in some other way?
>
> See above.
>
> > Are there other situations where the WAL is needed (or append
> > is needed) to avoid data loss?
>
> Deletes? Increments? For these operations you would not lose data per se if
> you don't have append support, but the client may be incorrectly led to
> believe they were successfully applied under the same low probability
> failure conditions that can corrupt META.
>
>  - Andy
>
>

> From: Andreas Neumann <[EMAIL PROTECTED]>

> I guess that includes region splits but also
> reassignment of a region after its region server died.

If you are not writing, then you won't see splits.

Reassignment does involve writes to META but missing these is less serious than missing a change in table structure. Stack could say more here but I believe the Master will discover inconsistency and repair it.

> But they could be rare enough to justify a less efficient
> implementation of the WAL. Would it be reasonable to use an
> implementation of HLog that - at the price of performance -
> persists the WAL to HDFS without relying on append?

How would that work if not create/write/close on every WAL commit? Correct me if I'm wrong, but there is no way to guarantee persistence in an HDFS without append except to .close() the file.

HBase can do the (near) equivalent of this without requiring any code changes. Set hbase.regionserver.logroll.period to an insanely low value.

   - Andy

> I agree that even though these are rare, they are not rare enough to take a
> risk. But they could be rare enough to justify a less efficient
> implementation of the WAL. Would it be reasonable to use an implementation
> of HLog that - at the price of performance - persists the WAL to HDFS
> without relying on append?

Since edits to .META. should be rare if you're using bulkImport, why
not just pay the cost of appending to the WAL? My guess is performance
would not suffer greatly. Would be nice to benchmark my claim :)

-Joey

On Wed, Jun 22, 2011 at 9:45 PM, Andreas Neumann <[EMAIL PROTECTED]> wrote:
> Thanks Andy for the clear response.
>
> We are indeed going to use bulk load only, and no puts, deletes or
> increments. So the only puts we will have are those that are caused by
> changes in the table structure. I guess that includes region splits but also
> reassignment of a region after its region server died.
>
> I agree that even though these are rare, they are not rare enough to take a
> risk. But they could be rare enough to justify a less efficient
> implementation of the WAL. Would it be reasonable to use an implementation
> of HLog that - at the price of performance - persists the WAL to HDFS
> without relying on append?
>
> Cheers -Andreas.
>
>
> On Wed, Jun 22, 2011 at 4:36 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
>
>> > From: Andreas Neumann <[EMAIL PROTECTED]>
>> > If we only load data in bulk (that is, via doBulkLoad(), not using
>> > TableOutputFormat), do we still risk data loss? My understanding is
>> > that append is needed for the WAL, and the WAL is needed only
>> > for puts. But bulk loads bypass the WAL.
>>
>> Correct.
>>
>> If you are doing read-only serving of HFiles built by MR and loaded by
>> doBulkLoad, then you would not need append support.
>>
>> If adding new data to tables via the HBase API, then sooner or later this
>> will change table structure, which is recorded via Puts to META, which is
>> self-hosted. Circumstances where those edits can be lost without working
>> append support in HDFS may be rare but not rare enough in my estimation.
>> Losing edits to META is bad. This can lead to missing regions and hung
>> clients. Human intervention will be necessary and the time scale for
>> administrative recovery is usually an availability problem.
>>
>> > For instance, when a region is split, the master must write
>> > the new meta data to the meta regions. Would that require a WAL
>> > or rely on append in some other way?
>>
>> See above.
>>
>> > Are there other situations where the WAL is needed (or append
>> > is needed) to avoid data loss?
>>
>> Deletes? Increments? For these operations you would not lose data per se if
>> you don't have append support, but the client may be incorrectly led to
>> believe they were successfully applied under the same low probability
>> failure conditions that can corrupt META.
>>
>>  - Andy
>>
>>
>

--
Joseph Echeverria
Cloudera, Inc.
443.305.9434

Andy,
we will use LoadIncrementalHFiles, are you saying that this will never cause
a split?

Joey,
our policies will not allow us to deploy a Hadoop version with append, until
there is an Apache release that includes both security and append. So
t.isis not about cost of appending to the WAL - we simply don't have
that
option.

Thanks -Andreas.

On Wed, Jun 22, 2011 at 7:14 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:

> > I agree that even though these are rare, they are not rare enough to take
> a
> > risk. But they could be rare enough to justify a less efficient
> > implementation of the WAL. Would it be reasonable to use an
> implementation
> > of HLog that - at the price of performance - persists the WAL to HDFS
> > without relying on append?
>
> Since edits to .META. should be rare if you're using bulkImport, why
> not just pay the cost of appending to the WAL? My guess is performance
> would not suffer greatly. Would be nice to benchmark my claim :)
>
> -Joey
>

On Wed, Jun 22, 2011 at 7:09 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> > From: Andreas Neumann <[EMAIL PROTECTED]>
>
> > I guess that includes region splits but also
> > reassignment of a region after its region server died.
>
> If you are not writing, then you won't see splits.
>
> Reassignment does involve writes to META but missing these is less serious
> than missing a change in table structure. Stack could say more here but I
> believe the Master will discover inconsistency and repair it.
>
> > But they could be rare enough to justify a less efficient
> > implementation of the WAL. Would it be reasonable to use an
> > implementation of HLog that - at the price of performance -
> > persists the WAL to HDFS without relying on append?
>
> How would that work if not create/write/close on every WAL commit? Correct
> me if I'm wrong, but there is no way to guarantee persistence in an HDFS
> without append except to .close() the file.
>
> HBase can do the (near) equivalent of this without requiring any code
> changes. Set hbase.regionserver.logroll.period to an insanely low value.
>
>   - Andy
>
>

On Wed, Jun 22, 2011 at 7:09 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
>> From: Andreas Neumann <[EMAIL PROTECTED]>
>
>> I guess that includes region splits but also
>> reassignment of a region after its region server died.
>
> If you are not writing, then you won't see splits.
>

You can also disable splits.
> Reassignment does involve writes to META but missing these is less serious than missing a change in table structure. Stack could say more here but I believe the Master will discover inconsistency and repair it.
>

Not really  In 0.20.x we had a thread that did fixup on .META. by
scanning it on a period and it notice if a region was assigned to a
regionserver that was no longer registered but we don't really do that
any more.  If it turns out you have a problem in here Andreas, you
could run the .META. checker externally and if an inconsistency, its
easy enough forcing region reassign.
> HBase can do the (near) equivalent of this without requiring any code changes. Set hbase.regionserver.logroll.period to an insanely low value.
>

As Andy says, this would be S L O W but your write rate might be low
enough -- only .META. edits -- it might be work (set log roll in the
kbs rather than mbs).

St.Ack

> From: Andreas Neumann <[EMAIL PROTECTED]>
> we will use LoadIncrementalHFiles, are you saying that this
> will never cause a split?

Create the table with the region split threshold set to Long.MAX_VALUE and a set of pre-split points that partitions the key space as evenly as possible.

Use HBase's TotalOrderPartitioner over row keys and HFileOutputFormat to build HFiles that fit within the predefined splits.

The result will never split.

If you do not modify the table(s), there will be no compaction activity either.

  - Andy

Actually I want compactions, and I think I also want splits, because the
data is going to grow over time.
So perhaps we should trigger splits and compaction manually... but I am
wondering whether it would be desirable that Hbase triggers them just as if
I load my data using puts. LoadIncrementalHFiles is just an alternative way
to achieve the loading of a large batch of data into HBase. But
semantically, shouldn't it have the  (or similar) effect on the table
as calling many individual puts?

Regarding the alternative HLog implementation, yes I was thinking of writing
and closing the file for every WAL write. But if the only writes are caused
by tables structure changes such as splits and reassignment, then that is
rare and the performance penalty could be tolerable. I was wondering whether
anybody had tried that before.

Thanks -Andreas.

On Thu, Jun 23, 2011 at 12:09 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:

> > From: Andreas Neumann <[EMAIL PROTECTED]>
> > we will use LoadIncrementalHFiles, are you saying that this
> > will never cause a split?
>
> Create the table with the region split threshold set to Long.MAX_VALUE and
> a set of pre-split points that partitions the key space as evenly as
> possible.
>
> Use HBase's TotalOrderPartitioner over row keys and HFileOutputFormat to
> build HFiles that fit within the predefined splits.
>
> The result will never split.
>
> If you do not modify the table(s), there will be no compaction activity
> either.
>
>  - Andy
>

> From: Andreas Neumann <[EMAIL PROTECTED]>
> Actually I want compactions

Fair enough but this doesn't involve META or append, I mentioned it as an aside.

> think I also want splits, because the data is going to grow
> over time.

Examine that assumption. If you have sufficient distribution of preallocated regions over the cluster (according to the space of keys of the application) then the accumulation of additional regions will probably just make things slower.

Anyway...

> Regarding the alternative HLog implementation, yes I was
> thinking of writing and closing the file for every WAL write.
> But if the only writes are caused by tables structure changes
> such as splits and reassignment

Modify HLog code to always roll after a META edit and you should be good.

   - Andy

Thanks, Andy, we will try that.
-Andreas.
On Thu, Jun 23, 2011 at 10:42 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:

> > From: Andreas Neumann <[EMAIL PROTECTED]>
> > Actually I want compactions
>
> Fair enough but this doesn't involve META or append, I mentioned it as an
> aside.
>
> > think I also want splits, because the data is going to grow
> > over time.
>
> Examine that assumption. If you have sufficient distribution of
> preallocated regions over the cluster (according to the space of keys of the
> application) then the accumulation of additional regions will probably just
> make things slower.
>
> Anyway...
>
> > Regarding the alternative HLog implementation, yes I was
> > thinking of writing and closing the file for every WAL write.
> > But if the only writes are caused by tables structure changes
> > such as splits and reassignment
>
> Modify HLog code to always roll after a META edit and you should be good.
>
>   - Andy
>
>
>