Hello,

I was reading an article on web which tells about MapReduce local Task and
use of hash table files and conditional tasks to improve performance of
hive queries.

Any idea how to implement this ? I am aware of Map joins but I am sure how
to implement Map reduce local tasks with hash tables.

Good wishes,always !
Santosh

Is setting hive.auto.convert.join to true will help setting mapreduce local
task and conditional task ?

Good wishes,always !
Santosh
On Tue, Jan 8, 2013 at 4:04 PM, Santosh Achhra <[EMAIL PROTECTED]>wrote:

> Hello,
>
> I was reading an article on web which tells about MapReduce local Task and
> use of hash table files and conditional tasks to improve performance of
> hive queries.
>
> Any idea how to implement this ? I am aware of Map joins but I am sure how
> to implement Map reduce local tasks with hash tables.
>
> Good wishes,always !
> Santosh
>

That setting will make Hive more aggressive about trying to convert a join
to a local task, where it bypasses the job tracker. When you're
experimenting with queries on a small data set, it can make things much
faster, but won't be useful for large data sets where you need the cluster.

dean

On Tue, Jan 8, 2013 at 9:11 AM, Santosh Achhra <[EMAIL PROTECTED]>wrote:

>
>
> Is setting hive.auto.convert.join to true will help setting mapreduce
> local task and conditional task ?
>
> Good wishes,always !
> Santosh
>
>
> On Tue, Jan 8, 2013 at 4:04 PM, Santosh Achhra <[EMAIL PROTECTED]>wrote:
>
>> Hello,
>>
>> I was reading an article on web which tells about MapReduce local Task
>> and use of hash table files and conditional tasks to improve performance of
>> hive queries.
>>
>> Any idea how to implement this ? I am aware of Map joins but I am sure
>> how to implement Map reduce local tasks with hash tables.
>>
>> Good wishes,always !
>> Santosh
>>
>
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330

Thank you Dean,

One of our table is very small, it has only 16,000 rows and other big table
has 45 million plus records. Wont doing a loacl task help in this case ?

Good wishes,always !
Santosh
On Tue, Jan 8, 2013 at 11:59 PM, Dean Wampler <
[EMAIL PROTECTED]> wrote:

> more aggressive about trying to convert a join to a local task, where it
> bypasses the job tracker. When you're experimenting with queries on a small
> data set, it can make things much faster, but won't be useful for large
> data sets where you need the cluster.
>

Hi Santhosh

As long as the smaller table size is in the range of a few MBs. It is a good candidate for map join.

If the smaller table size is still more then you can take a look at bucketed map joins.

Regards
Bejoy KS

Sent from remote device, Please excuse typos

-----Original Message-----
From: Santosh Achhra <[EMAIL PROTECTED]>
Date: Wed, 9 Jan 2013 00:11:37
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: Map Reduce Local Task

Thank you Dean,

One of our table is very small, it has only 16,000 rows and other big table
has 45 million plus records. Wont doing a loacl task help in this case ?

Good wishes,always !
Santosh
On Tue, Jan 8, 2013 at 11:59 PM, Dean Wampler <
[EMAIL PROTECTED]> wrote:

> more aggressive about trying to convert a join to a local task, where it
> bypasses the job tracker. When you're experimenting with queries on a small
> data set, it can make things much faster, but won't be useful for large
> data sets where you need the cluster.
>