Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Setting up a Hadoop client in OSGI bundle


Copy link to this message
-
Setting up a Hadoop client in OSGI bundle
Amit Sela 2013-04-17, 18:41
Hi all,

I'm trying to setup an Hadoop client for job submissions (and more) as an
OSGI bundle.

I came over a lot of hardships but I'm kinda stuck now.

When I create a new Job for submission I setClassLoader() for the Job
Configuration  so that it would use the bundle's ClassLoader (felix), but
when I submit the job and it tries to load the splits it looks for
DefaultCodec and fails to find that because somewhere along the way,
JobClient creates a new JobContext(Configuration) but although using the
Configuration containing the bundle's ClassLoader, the jobContext
constructor doesn't copy the ClassLoader set in Configuration.
And I guess that could happen in many other places in the Hadoop frameworl.

does anyone have any experience with submitting mapreduce jobs from OSGI
bundle client ? Any tips for this problems (and others I've yet to
encounter) ?

BTW, running Hadoop 1.0.4 and HBase 0.94.2.

Thanks,

Amit.