Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> loading Hadoop native libraries in HBase unit tests


Copy link to this message
-
Re: loading Hadoop native libraries in HBase unit tests
One difficulty with simply importing libhadoop.so into HBase codebase is
that the dynamic library is probably a bit different for different versions
of Hadoop. Is there a way to pull the .so file from Maven for the
configured Hadoop version? Ideally this should be done in a
platform-independent way, too, but making it work on Linux would be the
first step.

Thanks,
--Mikhail

On Fri, Feb 10, 2012 at 8:52 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Fri, Feb 10, 2012 at 7:22 PM, Mikhail Bautin
> <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> > We currently do not enable native Hadoop libraries in unit tests (at
> least
> > when running on Hadoop QA), but we do use them in production.  Should we
> > try to close this discrepancy between tests and production? Some possible
> > approaches would be:
> >
> >   - Enable native libraries by default (e.g. import libhadoop.so into a
> >   special location in the HBase repository and add
> >   -Djava.library.path=/that/path to unit test JVM options).
> >   - In addition to the above, modify some unit tests (e.g. those using
> >   Gzip compression) to run both with and without native libraries enabled
> >
> > Please let me know if this sounds reasonable.
> >
>
> We used to be careful w/ native libs to ensure they were present and
> hbase would use them if available but we let that drop a couple of
> versions ago and haven't picked it up since.  Thanks for reviving them
> Mikhail.  I can help making sure works on apache build box.
>
> St.Ack
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB