Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> RE: hive task fails when left semi join


+
Devaraj k 2013-07-16, 07:37
Copy link to this message
-
Re: hive task fails when left semi join
Dev,

from what I learned in my past exp with running huge one table queries is
one hits reduce side memory limits or timeout limits. I will wait for Kira
to give more details on the same.
sorry i forgot to ask for the logs and suggested a different approach :(

Kira,
Page is in chinese so can't make much out of it but the query looks like
map join.
If you are using older hive version
then the query showed on the mail thread looks good

if you are using new hive version then
 hive.auto.convert.join=true will do the job
On Tue, Jul 16, 2013 at 1:07 PM, Devaraj k <[EMAIL PROTECTED]> wrote:

>  Hi,****
>
> ****
>
>    In the given image, I see there are some failed/killed map& reduce task
> attempts. Could you check why these are failing, you can check further
> based on the fail/kill reason.****
>
> ** **
>
> ** **
>
> Thanks****
>
> Devaraj k****
>
> ** **
>
> *From:* [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
> *Sent:* 16 July 2013 12:57
> *To:* [EMAIL PROTECTED]
> *Subject:* hive task fails when left semi join****
>
> ** **
>
> Hello,****
>
> ** **
>
> I am trying to filter out some records in a table in hive.****
>
> The number of lines in this table is 4billions+, ****
>
> I make a left semi join between above table and a small table with 1k
> lines.****
>
> ** **
>
> However, after 3 hours job running, it turns out a fail status.****
>
> ** **
>
> My question are as follows,****
>
> **1.     **How could I address this problem and final solve it?****
>
> **2.     **Is there any other good methods could filter out records with
> give conditions?****
>
> ** **
>
> The following picture is a snapshot of the failed job.****
>
> ****
>
> ** **
>

--
Nitin Pawar
+
kira.wang@... 2013-07-16, 08:33
+
Nitin Pawar 2013-07-16, 08:43
+
kira.wang@... 2013-07-16, 09:05
+
kira.wang@... 2013-07-16, 07:49
+
kira.wang@... 2013-07-16, 07:36