Hi All,

I am trying to use hive inside an OSGi enviornment. So far I faced several
OSGi issues in hive and some issues related to JDO implementation used
(datanucleaus) under the OSGi enviornemnt and was able to overcome them.
However when I run a query involving a map reduce operation I am getting a
ClassNotFound exception from hadoop side. (Hadoop is running in pseudo
distributed mode in the local machine).

java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException:
org.apache.hadoop.hive.shims.HadoopShimsSecure$NullOutputCommitter
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
at org.apache.hadoop.mapred.JobConf.getOutputCommitter(JobConf.java:613)
at org.apache.hadoop.mapred.Task.initialize(Task.java:492)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352)
I think this is probably due to some jar/ resource not being bundled
properly in to the deployed archive to Hadoop. Would some one be able to
explain a bit of how the job submission part from Hive to Hadoop works and
any possible reason for this behaviour? Any pointers to look in to the code
is also appreciated.

Regards
Buddhika

HI Buddhika,

Which version of Hive are you running?

Thanks.

Carl

On Thu, Apr 5, 2012 at 10:12 AM, buddhika chamith
<[EMAIL PROTECTED]>wrote:

> Hi All,
>
> I am trying to use hive inside an OSGi enviornment. So far I faced several
> OSGi issues in hive and some issues related to JDO implementation used
> (datanucleaus) under the OSGi enviornemnt and was able to overcome them.
> However when I run a query involving a map reduce operation I am getting a
> ClassNotFound exception from hadoop side. (Hadoop is running in pseudo
> distributed mode in the local machine).
>
> java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.shims.HadoopShimsSecure$NullOutputCommitter
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
> at org.apache.hadoop.mapred.JobConf.getOutputCommitter(JobConf.java:613)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:492)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352)
>
>
> I think this is probably due to some jar/ resource not being bundled
> properly in to the deployed archive to Hadoop. Would some one be able to
> explain a bit of how the job submission part from Hive to Hadoop works and
> any possible reason for this behaviour? Any pointers to look in to the code
> is also appreciated.
>
> Regards
> Buddhika
>

Your hive jars in hadoop and restart the hadoop then try.

On Fri, Apr 6, 2012 at 12:49 AM, Carl Steinbach <[EMAIL PROTECTED]> wrote:

> HI Buddhika,
>
> Which version of Hive are you running?
>
> Thanks.
>
> Carl
>
> On Thu, Apr 5, 2012 at 10:12 AM, buddhika chamith <[EMAIL PROTECTED]
> > wrote:
>
>> Hi All,
>>
>> I am trying to use hive inside an OSGi enviornment. So far I faced
>> several OSGi issues in hive and some issues related to JDO implementation
>> used (datanucleaus) under the OSGi enviornemnt and was able to overcome
>> them. However when I run a query involving a map reduce operation I am
>> getting a ClassNotFound exception from hadoop side. (Hadoop is running in
>> pseudo distributed mode in the local machine).
>>
>> java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.shims.HadoopShimsSecure$NullOutputCommitter
>> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
>> at org.apache.hadoop.mapred.JobConf.getOutputCommitter(JobConf.java:613)
>> at org.apache.hadoop.mapred.Task.initialize(Task.java:492)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352)
>>
>>
>> I think this is probably due to some jar/ resource not being bundled
>> properly in to the deployed archive to Hadoop. Would some one be able to
>> explain a bit of how the job submission part from Hive to Hadoop works and
>> any possible reason for this behaviour? Any pointers to look in to the code
>> is also appreciated.
>>
>> Regards
>> Buddhika
>>
>
>
--
∞
Shashwat Shriparv

Hi,

Sorry I forgot to mention the versions. I am running Hive 0.8.1 and Hadoop
0.20.203.0. Shashwat's suggestion worked :). (I dropped hive-exec to hadoop
lib). Thanks. Anyway I am curious how this happened since with hive running
standalone (i mean not inside OSGi) this issue didn't come up. May be
hive-exec not getting uploaded to hdfs properly during hadoop job
submission?

Regards
Buddhika
On Fri, Apr 6, 2012 at 1:21 AM, shashwat shriparv <[EMAIL PROTECTED]
> wrote:

> Your hive jars in hadoop and restart the hadoop then try.
>
>
> On Fri, Apr 6, 2012 at 12:49 AM, Carl Steinbach <[EMAIL PROTECTED]> wrote:
>
>> HI Buddhika,
>>
>> Which version of Hive are you running?
>>
>> Thanks.
>>
>> Carl
>>
>> On Thu, Apr 5, 2012 at 10:12 AM, buddhika chamith <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi All,
>>>
>>> I am trying to use hive inside an OSGi enviornment. So far I faced
>>> several OSGi issues in hive and some issues related to JDO implementation
>>> used (datanucleaus) under the OSGi enviornemnt and was able to overcome
>>> them. However when I run a query involving a map reduce operation I am
>>> getting a ClassNotFound exception from hadoop side. (Hadoop is running in
>>> pseudo distributed mode in the local machine).
>>>
>>> java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.shims.HadoopShimsSecure$NullOutputCommitter
>>> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
>>> at org.apache.hadoop.mapred.JobConf.getOutputCommitter(JobConf.java:613)
>>> at org.apache.hadoop.mapred.Task.initialize(Task.java:492)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352)
>>>
>>>
>>> I think this is probably due to some jar/ resource not being bundled
>>> properly in to the deployed archive to Hadoop. Would some one be able to
>>> explain a bit of how the job submission part from Hive to Hadoop works and
>>> any possible reason for this behaviour? Any pointers to look in to the code
>>> is also appreciated.
>>>
>>> Regards
>>> Buddhika
>>>
>>
>>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>