Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> difference between add jar in hive session and hive --auxpath


Copy link to this message
-
RE: difference between add jar in hive session and hive --auxpath

This is in HIVE-0.9.0
hive> list jars;/nfs_home/common/userlibs/google-collections-1.0.jar/nfs_home/common/userlibs/elephant-bird-hive-3.0.7.jar/nfs_home/common/userlibs/protobuf-java-2.3.0.jar/nfs_home/common/userlibs/elephant-bird-core-3.0.7.jarfile:/usr/lib/hive/lib/hive-builtins-0.9.0-cdh4.1.2.jarhive> desc table;java.lang.NoClassDefFoundError: com/twitter/elephantbird/mapreduce/io/ProtobufConverter        at com.twitter.elephantbird.hive.serde.ProtobufDeserializer.initialize(ProtobufDeserializer.java:45)        at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:203)        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:260)        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:253)        at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:490)        at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:162)        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:930)        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:844)        at org.apache.hadoop.hive.ql.exec.DDLTask.describeTable(DDLTask.java:2545)        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:309)        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331)        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117)        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:744)        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)        at java.lang.reflect.Method.invoke(Method.java:597)        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)Caused by: java.lang.ClassNotFoundException: com.twitter.elephantbird.mapreduce.io.ProtobufConverter        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)        at java.security.AccessController.doPrivileged(Native Method)        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)        ... 25 moreFAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.DDLTaskhive> exit;[y130zhan@daca2 userlibs]$ jar tvf /nfs_home/common/userlibs/elephant-bird-core-3.0.7.jar | grep ProtobufConverter  4825 Mon Mar 04 16:50:46 UTC 2013 com/twitter/elephantbird/mapreduce/io/ProtobufConverter.class   732 Mon Mar 04 16:50:46 UTC 2013 com/twitter/elephantbird/mapreduce/io/ProtobufConverter$1.class

From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: RE: difference between add jar in hive session and hive --auxpath
Date: Thu, 7 Mar 2013 16:44:41 -0800
If properly done, "add jar <jar-file>" should work the same as passing the jar with --auxpath. Can you run "list jars;" command from CLI or Hue and check if you see the jar file.

From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: difference between add jar in hive session and hive --auxpath
Date: Thu, 7 Mar 2013 17:47:26 -0500

Hi,
I have a hive table which uses the jar file provided from the elephant-bird, which is a framework integrated between lzo and google protobuf data and hadoop/hive.
If I use the hive command like this:
hive --auxpath path_to_jars, it works fine to query my table,
but if I use the add jar after I started the hive session, I will get ClassNotFoundException in the runtime of my query of the classes in those jars.
My questions are:
1) What is the different between hive --auxpath and "add jar" in the hive session?2) This problem makes it is hard to access my table in the HUE, as it only supports "add jar", but not --auxpath option. Any suggestions?

Thanks
Yong            
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB