Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Creating Indexes


+
Peter Marron 2012-10-31, 17:42
Copy link to this message
-
Re: Creating Indexes
Hi Peter,

Can you attach the execution logs? What is the exception that you see in
the execution logs?

Thanks,
Shreepadma

On Wed, Oct 31, 2012 at 10:42 AM, Peter Marron <
[EMAIL PROTECTED]> wrote:

>  Hi,****
>
> ** **
>
> I am still having problems building my index.****
>
> In an attempt to find someone who can help me****
>
> I’ll go through all the steps that I try.****
>
> ** **
>
> **1)      **First I load my data into hive.****
>
> ** **
>
> hive> LOAD DATA INPATH 'E3/score.csv' OVERWRITE INTO TABLE score;****
>
> Loading data to table default.score****
>
> Deleted hdfs://localhost/data/warehouse/score****
>
> OK****
>
> Time taken: 7.817 seconds****
>
> ** **
>
> **2)      **Then I try to create the index ****
>
> ** **
>
> hive> CREATE INDEX bigIndex****
>
>     > ON TABLE score(Ath_Seq_Num)****
>
>     > AS 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler';***
> *
>
> FAILED: Error in metadata: java.lang.RuntimeException: Please specify
> deferred rebuild using " WITH DEFERRED REBUILD ".****
>
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask****
>
> hive> ****
>
> ** **
>
> **3)      **OK, so it suggests that I use “DEFERRED BUILD” and so I do****
>
> hive> ****
>
>     > ****
>
>     > CREATE INDEX bigIndex****
>
>     > ON TABLE score(Ath_Seq_Num)****
>
>     > AS 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler'****
>
>     > WITH DEFERRED REBUILD;****
>
> OK****
>
> Time taken: 0.603 seconds****
>
> ** **
>
> **4)      **Now, to create the index I assume that I use ALTER INDEX as
> follows:****
>
> ** **
>
> hive>ALTER INDEX bigIndex ON score REBUILD;****
>
> Total MapReduce jobs = 1****
>
> Launching Job 1 out of 1****
>
> Number of reduce tasks not specified. Estimated from input data size: 138*
> ***
>
> In order to change the average load for a reducer (in bytes):****
>
>   set hive.exec.reducers.bytes.per.reducer=<number>****
>
> In order to limit the maximum number of reducers:****
>
>   set hive.exec.reducers.max=<number>****
>
> In order to set a constant number of reducers:****
>
>   set mapred.reduce.tasks=<number>****
>
> Starting Job = job_201210311448_0001, Tracking URL > http://localhost:50030/jobdetails.jsp?jobid=job_201210311448_0001****
>
> Kill Command = /data/hadoop-1.0.3/libexec/../bin/hadoop job
> -Dmapred.job.tracker=localhost:8021 -kill job_201210311448_0001****
>
> Hadoop job information for Stage-1: number of mappers: 511; number of
> reducers: 138****
>
> 2012-10-31 15:59:27,076 Stage-1 map = 0%,  reduce = 0%****
>
> ** **
>
> **5)      **This all looks promising, and after increasing my heapsize to
> get the Map/Reduce to complete, I get this an hour later****
>
> ** **
>
> 2012-10-31 17:08:23,572 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU
> 4135.47 sec****
>
> MapReduce Total cumulative CPU time: 0 days 1 hours 8 minutes 55 seconds
> 470 msec****
>
> Ended Job = job_201210311448_0001****
>
> Loading data to table default.default__score_bigindex__****
>
> Deleted hdfs://localhost/data/warehouse/default__score_bigindex__****
>
> Invalid alter operation: Unable to alter index.****
>
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask****
>
> ** **
>
> So what have I done wrong, and what am I to do to get this index to build
> successfully?****
>
> ** **
>
> Any help appreciated.****
>
> ** **
>
> Peter Marron****
>
> ** **
>
> *From:* Peter Marron [mailto:[EMAIL PROTECTED]]
> *Sent:* 24 October 2012 13:27
> *To:* [EMAIL PROTECTED]
> *Subject:* RE: Indexes****
>
> ** **
>
> Hi Shreepadma,****
>
> ** **
>
> Thanks for this. Looks exactly like the information I need.****
>
> I was going to reply when I had tried it all out, but I’m having****
>
> problems creating the index at the moment (I’m getting an****
>
> OutOfMemoryError at the moment). So I thought that I had****
>
> better reply now to say thank you.****
>
> ** **
>
> Peter Marron****
>
> ** **
+
Peter Marron 2012-10-31, 19:52
+
Shreepadma Venugopalan 2012-10-31, 21:58
+
Peter Marron 2012-11-01, 09:31
+
Dean Wampler 2012-11-01, 13:01
+
Bejoy KS 2012-11-01, 13:09
+
Dean Wampler 2012-11-01, 13:20
+
Peter Marron 2012-11-02, 08:44
+
Dean Wampler 2012-11-02, 14:02
+
Peter Marron 2012-11-02, 23:59
+
Shreepadma Venugopalan 2012-11-03, 00:06
+
Peter Marron 2012-11-07, 13:33
+
Dean Wampler 2012-11-03, 00:02
+
Dean Wampler 2012-11-03, 00:03
+
Peter Marron 2012-11-23, 08:56