Hello,

In order to provide a custom "serialization.class" to a SerDe, I created a
jar containing all my custom serialization classes and added them to the
hive classpath with "ADD JAR my-classes.jar" command. Now when I try to use
these custom classes via CLI, it still throws me a "ClassNotFoundException"
for those custom classes in my jar.

Is there something that I am missing? I confirmed that 'list jars' is
showing me the custom jar that I added.

Any help would be appreciated.

Thanks,
--
Swarnim

You need to put these jars in your aux_lib folder or in your hadoop
classpath. There is a subtle difference between that classpath and the
classpath used by UDF and anything that involves  a serde or input
format needs to be in auxlib.

Edward

On Wed, Jun 13, 2012 at 10:20 AM, [EMAIL PROTECTED]
<[EMAIL PROTECTED]> wrote:
> Hello,
>
> In order to provide a custom "serialization.class" to a SerDe, I created a
> jar containing all my custom serialization classes and added them to the
> hive classpath with "ADD JAR my-classes.jar" command. Now when I try to use
> these custom classes via CLI, it still throws me a "ClassNotFoundException"
> for those custom classes in my jar.
>
> Is there something that I am missing? I confirmed that 'list jars' is
> showing me the custom jar that I added.
>
> Any help would be appreciated.
>
> Thanks,
> --
> Swarnim

Another option is to specify the path in the hive command:

hive --auxpath ~/my-classes.jar

-- Brad

On Jun 13, 2012, at 9:23 AM, Edward Capriolo wrote:

> You need to put these jars in your aux_lib folder or in your hadoop
> classpath. There is a subtle difference between that classpath and the
> classpath used by UDF and anything that involves  a serde or input
> format needs to be in auxlib.
>
> Edward
>
> On Wed, Jun 13, 2012 at 10:20 AM, [EMAIL PROTECTED]
> <[EMAIL PROTECTED]> wrote:
>> Hello,
>>
>> In order to provide a custom "serialization.class" to a SerDe, I created a
>> jar containing all my custom serialization classes and added them to the
>> hive classpath with "ADD JAR my-classes.jar" command. Now when I try to use
>> these custom classes via CLI, it still throws me a "ClassNotFoundException"
>> for those custom classes in my jar.
>>
>> Is there something that I am missing? I confirmed that 'list jars' is
>> showing me the custom jar that I added.
>>
>> Any help would be appreciated.
>>
>> Thanks,
>> --
>> Swarnim

Right, The end result is that same. The hive shell script currently
bulds a list of aux_lib jars and generates the auxpath arguments. but
it is good to know these files do not need to be in a static folder.

On Wed, Jun 13, 2012 at 10:32 AM, Rubin, Bradley S.
<[EMAIL PROTECTED]> wrote:
> Another option is to specify the path in the hive command:
>
> hive --auxpath ~/my-classes.jar
>
> -- Brad
>
> On Jun 13, 2012, at 9:23 AM, Edward Capriolo wrote:
>
>> You need to put these jars in your aux_lib folder or in your hadoop
>> classpath. There is a subtle difference between that classpath and the
>> classpath used by UDF and anything that involves  a serde or input
>> format needs to be in auxlib.
>>
>> Edward
>>
>> On Wed, Jun 13, 2012 at 10:20 AM, [EMAIL PROTECTED]
>> <[EMAIL PROTECTED]> wrote:
>>> Hello,
>>>
>>> In order to provide a custom "serialization.class" to a SerDe, I created a
>>> jar containing all my custom serialization classes and added them to the
>>> hive classpath with "ADD JAR my-classes.jar" command. Now when I try to use
>>> these custom classes via CLI, it still throws me a "ClassNotFoundException"
>>> for those custom classes in my jar.
>>>
>>> Is there something that I am missing? I confirmed that 'list jars' is
>>> showing me the custom jar that I added.
>>>
>>> Any help would be appreciated.
>>>
>>> Thanks,
>>> --
>>> Swarnim
>

Cool. That worked!

Thanks guys.

On Wed, Jun 13, 2012 at 9:35 AM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> Right, The end result is that same. The hive shell script currently
> bulds a list of aux_lib jars and generates the auxpath arguments. but
> it is good to know these files do not need to be in a static folder.
>
> On Wed, Jun 13, 2012 at 10:32 AM, Rubin, Bradley S.
> <[EMAIL PROTECTED]> wrote:
> > Another option is to specify the path in the hive command:
> >
> > hive --auxpath ~/my-classes.jar
> >
> > -- Brad
> >
> > On Jun 13, 2012, at 9:23 AM, Edward Capriolo wrote:
> >
> >> You need to put these jars in your aux_lib folder or in your hadoop
> >> classpath. There is a subtle difference between that classpath and the
> >> classpath used by UDF and anything that involves  a serde or input
> >> format needs to be in auxlib.
> >>
> >> Edward
> >>
> >> On Wed, Jun 13, 2012 at 10:20 AM, [EMAIL PROTECTED]
> >> <[EMAIL PROTECTED]> wrote:
> >>> Hello,
> >>>
> >>> In order to provide a custom "serialization.class" to a SerDe, I
> created a
> >>> jar containing all my custom serialization classes and added them to
> the
> >>> hive classpath with "ADD JAR my-classes.jar" command. Now when I try
> to use
> >>> these custom classes via CLI, it still throws me a
> "ClassNotFoundException"
> >>> for those custom classes in my jar.
> >>>
> >>> Is there something that I am missing? I confirmed that 'list jars' is
> >>> showing me the custom jar that I added.
> >>>
> >>> Any help would be appreciated.
> >>>
> >>> Thanks,
> >>> --
> >>> Swarnim
> >
>

--
Swarnim

Another quick follow up on this itself.

If I try to run a query that invokes a M/R job, then it throws me this
exception:

java.io.FileNotFoundException: File does not exist: /Users/my-classes.jar
 *at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:722)
*
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
 at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:77)

Looking at the stacktrace, it seems like it is searching for the jar on
HDFS rather than local filesystem. Is that intended? All "Select *" queries
that do not spawn a M/R job work fine.

Thanks,

On Wed, Jun 13, 2012 at 9:44 AM, [EMAIL PROTECTED] <
[EMAIL PROTECTED]> wrote:

> Cool. That worked!
>
> Thanks guys.
>
>
> On Wed, Jun 13, 2012 at 9:35 AM, Edward Capriolo <[EMAIL PROTECTED]>wrote:
>
>> Right, The end result is that same. The hive shell script currently
>> bulds a list of aux_lib jars and generates the auxpath arguments. but
>> it is good to know these files do not need to be in a static folder.
>>
>> On Wed, Jun 13, 2012 at 10:32 AM, Rubin, Bradley S.
>> <[EMAIL PROTECTED]> wrote:
>> > Another option is to specify the path in the hive command:
>> >
>> > hive --auxpath ~/my-classes.jar
>> >
>> > -- Brad
>> >
>> > On Jun 13, 2012, at 9:23 AM, Edward Capriolo wrote:
>> >
>> >> You need to put these jars in your aux_lib folder or in your hadoop
>> >> classpath. There is a subtle difference between that classpath and the
>> >> classpath used by UDF and anything that involves  a serde or input
>> >> format needs to be in auxlib.
>> >>
>> >> Edward
>> >>
>> >> On Wed, Jun 13, 2012 at 10:20 AM, [EMAIL PROTECTED]
>> >> <[EMAIL PROTECTED]> wrote:
>> >>> Hello,
>> >>>
>> >>> In order to provide a custom "serialization.class" to a SerDe, I
>> created a
>> >>> jar containing all my custom serialization classes and added them to
>> the
>> >>> hive classpath with "ADD JAR my-classes.jar" command. Now when I try
>> to use
>> >>> these custom classes via CLI, it still throws me a
>> "ClassNotFoundException"
>> >>> for those custom classes in my jar.
>> >>>
>> >>> Is there something that I am missing? I confirmed that 'list jars' is
>> >>> showing me the custom jar that I added.
>> >>>
>> >>> Any help would be appreciated.
>> >>>
>> >>> Thanks,
>> >>> --
>> >>> Swarnim
>> >
>>
>
>
>
> --
> Swarnim
>

--
Swarnim