Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Creating Indexes


Copy link to this message
-
RE: Creating Indexes
Peter Marron 2012-11-01, 09:31
Hi Shreepadma,

I agree that the error looks odd. However I can't believe that I would have
got this far with Hive if there was no derby jar. Nevertheless I checked.
Here is a directory listing of the Hive install:

pmarron@pmarron-ubuntu:/data/hive/lib$ ls
ant-contrib-1.0b3.jar          commons-pool-1.5.4.jar                hive-common-0.8.1.jar         hive-shims-0.8.1.jar  mockito-all-1.8.2.jar
antlr-2.7.7.jar                datanucleus-connectionpool-2.0.3.jar  hive-contrib-0.8.1.jar        javaewah-0.3.jar      php
antlr-3.0.1.jar                datanucleus-core-2.0.3.jar            hive_contrib.jar              jdo2-api-2.3-ec.jar   py
antlr-runtime-3.0.1.jar        datanucleus-enhancer-2.0.3.jar        hive-exec-0.8.1.jar           jline-0.9.94.jar      slf4j-api-1.6.1.jar
asm-3.1.jar                    datanucleus-rdbms-2.0.3.jar           hive-hbase-handler-0.8.1.jar  json-20090211.jar     slf4j-log4j12-1.6.1.jar
commons-cli-1.2.jar            derby-10.4.2.0.jar                    hive-hwi-0.8.1.jar            junit-4.10.jar        stringtemplate-3.1-b1.jar
commons-codec-1.3.jar          guava-r06.jar                         hive-hwi-0.8.1.war            libfb303-0.7.0.jar    velocity-1.5.jar
commons-collections-3.2.1.jar  hbase-0.89.0-SNAPSHOT.jar             hive-jdbc-0.8.1.jar           libfb303.jar          zookeeper-3.3.1.jar
commons-dbcp-1.4.jar           hbase-0.89.0-SNAPSHOT-tests.jar       hive-metastore-0.8.1.jar      libthrift-0.7.0.jar
commons-lang-2.4.jar           hive-anttasks-0.8.1.jar               hive-pdk-0.8.1.jar            libthrift.jar
commons-logging-1.0.4.jar      hive-builtins-0.8.1.jar               hive-serde-0.8.1.jar          log4j-1.2.15.jar
commons-logging-api-1.0.4.jar  hive-cli-0.8.1.jar                    hive-service-0.8.1.jar        log4j-1.2.16.jar

Also I found a derby.log in my home directory which I have attached.

Regards,

Z

From: Shreepadma Venugopalan [mailto:[EMAIL PROTECTED]]
Sent: 31 October 2012 21:58
To: [EMAIL PROTECTED]
Subject: Re: Creating Indexes

Hi Peter,

>From the execution log,

java.lang.ClassNotFoundException: org.apache.derby.jdbc.EmbeddedDriver
          at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
          at java.lang.Class.forName0(Native Method)
          at java.lang.Class.forName(Class.java:186)
          at org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.connect(JDBCStatsPublisher.java:68)
          at org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:778)
          at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:723)
          at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:557)
          at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
          at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
          at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:303)
          at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:529)
          at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
          at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:415)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
          at org.apache.hadoop.mapred.Child.main(Child.java:249)

It appears that the error is due derby classes not being found. Can you check if the derby jars are present?

Thanks,
Shreepadma
On Wed, Oct 31, 2012 at 12:52 PM, Peter Marron <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Shreepadma

Happy to attach the logs, not quite sure which one is going to
be most useful. Please find attached one which contained an
error of some sort. Not sure it it's related or not to the index error.
Found the file in this location:

/data/hadoop/logs/userlogs/job_201210311448_0001/attempt_201210311448_0001_r_000137_0/syslog

so maybe that will help you locate any other file that you might want to see.

Thanks for your efforts.

Peter Marron

From: Shreepadma Venugopalan [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: 31 October 2012 18:38
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Creating Indexes

Hi Peter,

Can you attach the execution logs? What is the exception that you see in the execution logs?

Thanks,
Shreepadma

On Wed, Oct 31, 2012 at 10:42 AM, Peter Marron <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi,

I am still having problems building my index.
In an attempt to find someone who can help me
I'll go through all the steps that I try.
1)      First I load my data into hive.

hive> LOAD DATA INPATH 'E3/score.csv' OVERWRITE INTO TABLE score;
Loading data to table default.score
Deleted hdfs://localhost/data/warehouse/score
OK
Time taken: 7.817 seconds
2)      Then I try to create the index

hive> CREATE INDEX bigIndex
    > ON TABLE score(Ath_Seq_Num)
    > AS 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler';
FAILED: Error in metadata: java.lang.RuntimeException: Please specify deferred rebuild using " WITH DEFERRED REBUILD ".
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive>
3)      OK, so it suggests that I use "DEFERRED BUILD" and so I do
hive>
    >
    > CREATE INDEX bigIndex
    > ON TABLE score(Ath_Seq_Num)
    > AS 'org.apache.hadoo