Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> difference between add jar in hive session and hive --auxpath


Copy link to this message
-
RE: difference between add jar in hive session and hive --auxpath

This is in HIVE-0.9.0
hive> list jars;/nfs_home/common/userlibs/google-collections-1.0.jar/nfs_home/common/userlibs/elephant-bird-hive-3.0.7.jar/nfs_home/common/userlibs/protobuf-java-2.3.0.jar/nfs_home/common/userlibs/elephant-bird-core-3.0.7.jarfile:/usr/lib/hive/lib/hive-builtins-0.9.0-cdh4.1.2.jarhive> desc table;java.lang.NoClassDefFoundError: com/twitter/elephantbird/mapreduce/io/ProtobufConverter        at com.twitter.elephantbird.hive.serde.ProtobufDeserializer.initialize(ProtobufDeserializer.java:45)        at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:203)        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:260)        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:253)        at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:490)        at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:162)        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:930)        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:844)        at org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:2545)        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:309)        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331)        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117)        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:744)        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)        at java.lang.reflect.Method.invoke(Method.java:597)        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)Caused by: java.lang.ClassNotFoundException: com.twitter.elephantbird.mapreduce.io.ProtobufConverter        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)        at java.security.AccessController.doPrivileged(Native Method)        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)        ... 25 moreFAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.DDLTaskhive> exit;[y130zhan@daca2 userlibs]$ jar tvf /nfs_home/common/userlibs/elephant-bird-core-3.0.7.jar | grep ProtobufConverter  4825 Mon Mar 04 16:50:46 UTC 2013 com/twitter/elephantbird/mapreduce/io/ProtobufConverter.class   732 Mon Mar 04 16:50:46 UTC 2013 com/twitter/elephantbird/mapreduce/io/ProtobufConverter$1.class

From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: RE: difference between add jar in hive session and hive --auxpath
Date: Thu, 7 Mar 2013 16:44:41 -0800
If properly done, "add jar <jar-file>" should work the same as passing the jar with --auxpath. Can you run "list jars;" command from CLI or Hue and check if you see the jar file.

From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: difference between add jar in hive session and hive --auxpath
Date: Thu, 7 Mar 2013 17:47:26 -0500

Hi,
I have a hive table which uses the jar file provided from the elephant-bird, which is a framework integrated between lzo and google protobuf data and hadoop/hive.
If I use the hive command like this:
hive --auxpath path_to_jars, it works fine to query my table,
but if I use the add jar after I started the hive session, I will get ClassNotFoundException in the runtime of my query of the classes in those jars.
My questions are:
1) What is the different between hive --auxpath and "add jar" in the hive session?2) This problem makes it is hard to access my table in the HUE, as it only supports "add jar", but not --auxpath option. Any suggestions?

Thanks
Yong