Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Creating Indexes


+
Peter Marron 2012-10-31, 17:42
+
Shreepadma Venugopalan 2012-10-31, 18:37
+
Peter Marron 2012-10-31, 19:52
+
Shreepadma Venugopalan 2012-10-31, 21:58
+
Peter Marron 2012-11-01, 09:31
+
Dean Wampler 2012-11-01, 13:01
+
Bejoy KS 2012-11-01, 13:09
+
Dean Wampler 2012-11-01, 13:20
+
Peter Marron 2012-11-02, 08:44
+
Dean Wampler 2012-11-02, 14:02
+
Peter Marron 2012-11-02, 23:59
+
Shreepadma Venugopalan 2012-11-03, 00:06
+
Peter Marron 2012-11-07, 13:33
+
Dean Wampler 2012-11-03, 00:02
Copy link to this message
-
Re: Creating Indexes
Hit send too soon...

I'm glad the ADD JAR hack appeared to work. You might verify if the
temporary files mentioned are still there and also verify that you have
write permissions for the target index directories. Other than that, I'm
not sure what to suggest. I haven't really used indexing much, but it
appears to be an immature feature, unfortunately.

dean

On Fri, Nov 2, 2012 at 7:02 PM, Dean Wampler <
[EMAIL PROTECTED]> wrote:

> Wow. Lots of quirks. I'm glad the ADD JAR
>
>
> On Fri, Nov 2, 2012 at 6:59 PM, Peter Marron <
> [EMAIL PROTECTED]> wrote:
>
>>  Hi Dean,****
>>
>> ** **
>>
>> At this stage I’m really not worried about this being a hack.****
>>
>> I just want to get it to work, and I’m grateful for all your help.****
>>
>> I did as you suggested and now, as far as I can see, the Map/Reduce****
>>
>> has succeeded. When I look in the log for the last reduce I no longer****
>>
>> find an error. However this is the output from the hive command****
>>
>> session:****
>>
>> ** **
>>
>> MapReduce Total cumulative CPU time: 0 days 1 hours 14 minutes 51 seconds
>> 360 msec****
>>
>> Ended Job = job_201211021743_0001****
>>
>> Loading data to table default.default__score_bigindex__****
>>
>> Deleted hdfs://localhost/data/warehouse/default__score_bigindex__****
>>
>> Invalid alter operation: Unable to alter index.****
>>
>> Table default.default__score_bigindex__ stats: [num_partitions: 0,
>> num_files: 138, num_rows: 0, total_size: 446609024, raw_data_size: 0]****
>>
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.DDLTask****
>>
>> MapReduce Jobs Launched: ****
>>
>> Job 0: Map: 511  Reduce: 138   Accumulative CPU: 4491.36 sec   HDFS Read:
>> 137123460712 HDFS Write: 446609024 SUCESS****
>>
>> Total MapReduce CPU Time Spent: 0 days 1 hours 14 minutes 51 seconds 360
>> msec****
>>
>> hive>     ****
>>
>> ** **
>>
>> I find this very confusing. We have the bit where it says “Job 0:….
>> SUCCESS”****
>>
>> and this seems to fit with the fact that I can’t find errors in the
>> Map/Reduce.****
>>
>> On the other hand we have the bit where it says: “Invalid alter
>> operation: Unable to alter index.”****
>>
>> So has it successfully created the index  or not? And if not, then what
>> do I do next?****
>>
>> Is there somewhere else where it records Hive errors as opposed to
>> Map/Reduce errors?****
>>
>> ** **
>>
>> Regards,****
>>
>> ** **
>>
>> Peter Marron****
>>
>>                                    ****
>>
>> ** **
>>
>> *From:* Dean Wampler [mailto:[EMAIL PROTECTED]]
>> *Sent:* 02 November 2012 14:03
>>
>> *To:* [EMAIL PROTECTED]
>> *Subject:* Re: Creating Indexes****
>>
>>  ** **
>>
>> Oh, I saw this line in your Hive output and just assumed you were running
>> in a cluster:****
>>
>> ** **
>>
>> Hadoop job information for Stage-1: number of mappers: 511; number of
>> reducers: 138****
>>
>> ** **
>>
>> I haven't tried running a job that big in pseudodistributed mode either,
>> but that's beside the point.****
>>
>> ** **
>>
>> So it seems to be an issue with indexing, but it still begs the question
>> why derby isn't on the classpath for the task. I would try using the ADD
>> JAR command, which copies the jar around the "cluster" and puts it on the
>> classpath. It's what you would use with UDFs, for example:****
>>
>> ** **
>>
>> ADD JAR /path/to/derby.jar****
>>
>> ALTER INDEX ...;****
>>
>> ** **
>>
>> It's a huge hack, but it just might work.****
>>
>> dean****
>>
>> ** **
>>
>> On Fri, Nov 2, 2012 at 3:44 AM, Peter Marron <
>> [EMAIL PROTECTED]> wrote:****
>>
>> Hi Dean,****
>>
>>  ****
>>
>> I’m running everything on a single physical machine in pseudo-distributed
>> mode.****
>>
>>  ****
>>
>> Well it certainly looks like the reducer is looking for a derby.jar,
>> although I must****
>>
>> confess I don’t really understand why it would be doing that.****
>>
>> In an effort to fix that I copied the derby.jar (derby-10.4.2.0.jar) into
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
+
Peter Marron 2012-11-23, 08:56
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB