Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - loading Hadoop native libraries in HBase unit tests


Copy link to this message
-
Re: loading Hadoop native libraries in HBase unit tests
Mikhail Bautin 2012-02-12, 20:36
One difficulty with simply importing libhadoop.so into HBase codebase is
that the dynamic library is probably a bit different for different versions
of Hadoop. Is there a way to pull the .so file from Maven for the
configured Hadoop version? Ideally this should be done in a
platform-independent way, too, but making it work on Linux would be the
first step.

Thanks,
--Mikhail

On Fri, Feb 10, 2012 at 8:52 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Fri, Feb 10, 2012 at 7:22 PM, Mikhail Bautin
> <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> > We currently do not enable native Hadoop libraries in unit tests (at
> least
> > when running on Hadoop QA), but we do use them in production.  Should we
> > try to close this discrepancy between tests and production? Some possible
> > approaches would be:
> >
> >   - Enable native libraries by default (e.g. import libhadoop.so into a
> >   special location in the HBase repository and add
> >   -Djava.library.path=/that/path to unit test JVM options).
> >   - In addition to the above, modify some unit tests (e.g. those using
> >   Gzip compression) to run both with and without native libraries enabled
> >
> > Please let me know if this sounds reasonable.
> >
>
> We used to be careful w/ native libs to ensure they were present and
> hbase would use them if available but we let that drop a couple of
> versions ago and haven't picked it up since.  Thanks for reviving them
> Mikhail.  I can help making sure works on apache build box.
>
> St.Ack
>