Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Providing a custom serialization.class SerDe property


Copy link to this message
-
Re: Providing a custom serialization.class SerDe property
Another quick follow up on this itself.

If I try to run a query that invokes a M/R job, then it throws me this
exception:

java.io.FileNotFoundException: File does not exist: /Users/my-classes.jar
 *at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:722)
*
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
 at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:77)

Looking at the stacktrace, it seems like it is searching for the jar on
HDFS rather than local filesystem. Is that intended? All "Select *" queries
that do not spawn a M/R job work fine.

Thanks,

On Wed, Jun 13, 2012 at 9:44 AM, [EMAIL PROTECTED] <
[EMAIL PROTECTED]> wrote:

> Cool. That worked!
>
> Thanks guys.
>
>
> On Wed, Jun 13, 2012 at 9:35 AM, Edward Capriolo <[EMAIL PROTECTED]>wrote:
>
>> Right, The end result is that same. The hive shell script currently
>> bulds a list of aux_lib jars and generates the auxpath arguments. but
>> it is good to know these files do not need to be in a static folder.
>>
>> On Wed, Jun 13, 2012 at 10:32 AM, Rubin, Bradley S.
>> <[EMAIL PROTECTED]> wrote:
>> > Another option is to specify the path in the hive command:
>> >
>> > hive --auxpath ~/my-classes.jar
>> >
>> > -- Brad
>> >
>> > On Jun 13, 2012, at 9:23 AM, Edward Capriolo wrote:
>> >
>> >> You need to put these jars in your aux_lib folder or in your hadoop
>> >> classpath. There is a subtle difference between that classpath and the
>> >> classpath used by UDF and anything that involves  a serde or input
>> >> format needs to be in auxlib.
>> >>
>> >> Edward
>> >>
>> >> On Wed, Jun 13, 2012 at 10:20 AM, [EMAIL PROTECTED]
>> >> <[EMAIL PROTECTED]> wrote:
>> >>> Hello,
>> >>>
>> >>> In order to provide a custom "serialization.class" to a SerDe, I
>> created a
>> >>> jar containing all my custom serialization classes and added them to
>> the
>> >>> hive classpath with "ADD JAR my-classes.jar" command. Now when I try
>> to use
>> >>> these custom classes via CLI, it still throws me a
>> "ClassNotFoundException"
>> >>> for those custom classes in my jar.
>> >>>
>> >>> Is there something that I am missing? I confirmed that 'list jars' is
>> >>> showing me the custom jar that I added.
>> >>>
>> >>> Any help would be appreciated.
>> >>>
>> >>> Thanks,
>> >>> --
>> >>> Swarnim
>> >
>>
>
>
>
> --
> Swarnim
>

--
Swarnim
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB