Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - DISCUSS: Have hbase require at least hadoop 1.0.0 in hbase 0.96.0?


+
Stack 2012-03-02, 19:24
+
Andrew Purtell 2012-03-03, 02:39
+
Ioan Eugen Stan 2012-03-06, 12:04
+
Huanyou Chang 2012-03-06, 10:57
+
Ramkrishna.S.Vasudevan 2012-03-06, 11:22
+
Ted Yu 2012-03-02, 20:50
+
Nicolas Spiegelberg 2012-03-02, 20:57
+
Stack 2012-03-06, 07:59
+
Andrew Purtell 2012-03-06, 17:10
+
Stack 2012-03-06, 17:55
+
Andrew Purtell 2012-03-06, 18:21
+
Arun C Murthy 2012-03-06, 19:57
+
Devaraj Das 2012-03-07, 18:20
Copy link to this message
-
Re: DISCUSS: Have hbase require at least hadoop 1.0.0 in hbase 0.96.0?
Mikhail Bautin 2012-03-07, 19:42
The current support for multiple versions of HDFS is in my opinion actually
one of the strengths of HBase, and the project will lose that advantage if
we cut support for earlier versions of Hadoop. I think HBase should only
require the simplest possible universally available subset of HDFS API, and
security should be an optional feature, discovered through reflection or
enabled in some other ways.

We have a custom version of Hadoop at Facebook that is not planning to
implement security any time soon. This version of Hadoop runs underneath
what we believe to be some of the largest existing production HBase
deployments. We are currently running the 0.89-fb version of HBase in
production, but are considering moving to a more recent version of HBase at
some point, and it would be great to be able to do that independently of
changing the underlying Hadoop distribution for migration complexity
reasons. Currently we are able to run public HBase trunk on our version of
Hadoop, but once in a while we have to satisfy new dependences on Hadoop
features that are added to HBase. If the changes proposed in this thread
happen, we would have to pull in a lot more security-related dependencies
into our version of Hadoop and, most likely, implement a lot of no-op
stubs. However, that may not be a trivial project, and it certainly would
not add any clarity or value to our Hadoop codebase or HBase / HDFS
interaction.

I imagine there are other custom flavors of Hadoop out there where HBase
support would be desirable. For example, does MapR implement the same
security API as Hadoop 1.0.0 does? Restricting HBase to a smaller subset of
Hadoop versions complicates life for existing users, and makes HBase a less
likely choice for new users, who could go with something like Hypertable
where they have an extra abstraction layer between the database and the
underlying distributed file system implementation.

Thanks,
--Mikhail

On Wed, Mar 7, 2012 at 10:20 AM, Devaraj Das <[EMAIL PROTECTED]> wrote:

> Given that the token/ugi APIs are being used in other ecosystem components
> too (like Hive, HCatalog & Oozie), and in general, that security model will
> probably hold for other projects too, I think that its not an unfair
> expectation from Hadoop that it should maintain compatibility on UGI/Token*
> interfaces (*smile*).
>
> On Mar 6, 2012, at 11:57 AM, Arun C Murthy wrote:
>
> > Andy - could you please start a discussion?
> >
> > We could, at the very least, mark UGI as LimitedPrivate for HBase and
> work with you guys to maintain compatibility for the future. Makes sense?
> >
> > thanks,
> > Arun
> >
> > On Mar 6, 2012, at 10:21 AM, Andrew Purtell wrote:
> >
> >> After that, I believe we can merge the security sources in. However we
> may have an issue going forward because UGI is an unstable/private API.
> Needs sorting out with core at some point.
> >>
> >> Best regards,
> >>
> >>   - Andy
> >>
> >>
> >> On Mar 6, 2012, at 9:55 AM, Stack <[EMAIL PROTECTED]> wrote:
> >>
> >>> On Tue, Mar 6, 2012 at 9:10 AM, Andrew Purtell <[EMAIL PROTECTED]>
> wrote:
> >>>> ...however we can't easily build a single artifact because the secure
> RPC engine, as it interacts with the Hadoop auth framework, must use
> UserGroupInformation.
> >>>>
> >>>
> >>> OK.  So security story needs a bit of work.  Sounds like we have
> >>> enough votes though to require hadoop 1.0.0 at least in 0.96.
> >>>
> >>> St.Ack
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>
>
+
Ted Yu 2012-03-07, 20:17
+
Andrew Purtell 2012-03-07, 22:39
+
Stack 2012-03-08, 06:09
+
Jonathan Hsieh 2012-03-08, 08:31
+
Stack 2012-03-08, 17:08
+
Mikhail Bautin 2012-03-08, 23:29
+
Andrew Purtell 2012-03-07, 22:32
+
Gary Helmling 2012-03-07, 09:38
+
lars hofhansl 2012-03-02, 19:29
+
Jesse Yates 2012-03-02, 19:32