Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Re: Map join optimization issue


Copy link to this message
-
Re: Map join optimization issue
bejoy_ks@... 2013-02-15, 16:57
Hi

In later versions of hive you actually don't need a map joint hint in your query. Just the following would suffice the purpose

Set hive.auto.convert.join=true

Regards
Bejoy KS

Sent from remote device, Please excuse typos

-----Original Message-----
From: Mayuresh Kunjir <[EMAIL PROTECTED]>
Date: Fri, 15 Feb 2013 10:37:52
To: user<[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: Map join optimization issue

Thanks Aniket. I actually had not specified the map-join hint though. Sorry
for providing the wrong information earlier. I had only
set hive.auto.convert.join=true before firing my join query.

~Mayuresh

On Thu, Feb 14, 2013 at 10:44 PM, Aniket Mokashi <[EMAIL PROTECTED]>wrote:

> I think hive.mapjoin.smalltable.filesize parameter will be disregarded in
> that case.
>
>
> On Thu, Feb 14, 2013 at 7:25 AM, Mayuresh Kunjir <
> [EMAIL PROTECTED]> wrote:
>
>> Yes, the hint was specified.
>> On Feb 14, 2013 3:11 AM, "Aniket Mokashi" <[EMAIL PROTECTED]> wrote:
>>
>>> have you specified map-join hint in your query?
>>>
>>>
>>> On Thu, Feb 7, 2013 at 11:39 AM, Mayuresh Kunjir <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>>
>>>> Hello all,
>>>>
>>>>
>>>> I am trying to join two tables, the smaller being of size 4GB. When I
>>>> set hive.mapjoin.smalltable.filesize parameter above 500MB, Hive tries to
>>>> perform a local task to read the smaller file. This of-course fails since
>>>> the file size is greater and the backup common join is then run. What I do
>>>> not understand is why did Hive attempt a map join when small file size was
>>>> greater than the smalltable.filesize parameter.
>>>>
>>>>
>>>> ~Mayuresh
>>>>
>>>>
>>>
>>>
>>> --
>>> "...:::Aniket:::... Quetzalco@tl"
>>>
>>
>
>
> --
> "...:::Aniket:::... Quetzalco@tl"
>