Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - Reduce operation stuck
Balaji Rao
2012-05-18, 18:57
Ranjith
2012-05-19, 03:50
Balaji Rao
2012-05-20, 16:36
Abhishek Pratap Singh
2012-05-21, 21:23
+
Balaji Rao 2012-05-18, 18:57
+
Ranjith 2012-05-19, 03:50
Copy link to this message
-
Re: Reduce operation stuck
Bejoy KS 2012-05-19, 04:15
Hi Balaji

How many reduce tasks are being triggered?
If more than 1, how many of them are completed and and how many are still running?
Can you try increasing the number of reduce tasks and post in the behavior you are seeing.

I believe skew is already enabled for joins in your hive configuration.

 
Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: Ranjith <[EMAIL PROTECTED]>
Date: Fri, 18 May 2012 22:50:47
To: [EMAIL PROTECTED]<[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]<[EMAIL PROTECTED]>
Subject: Re: Reduce operation stuck

This could be a result of the data being skewed. When you look at the job tracker page are you noticing just a few lurking around?

Thanks,
Ranjith

On May 18, 2012, at 1:57 PM, Balaji Rao <[EMAIL PROTECTED]> wrote:

> I have a simple query running on HIVE using Amazon EMR where the
> reduce operation seems to be stuck. The table is small with just
> 300,000+ rows (data_analysis).
> I intend to work on a 30,000,000 row table once I'm able to fix this.
> The query (modified to hide some information) is as follows:
>
> INSERT OVERWRITE TABLE data_analysis
> SELECT
>  a.clientid,  a.isi,  a.os,  a.bro,  a.engaged,  a.iframed,
>  hover, hower1, hower5, hower60,
>  Visible,  a.x1,  a.x2,  a.x3,
>  NoViewMeasure,  a.x4,  a.x5,  a.x6,
>  a.alerttype,    a.x7,  a.x9,  a.x11,  a.x13,  a.x17,  a.x21,  a.x27,  a.x28,
>  COUNT(a.auction_id_64),  SUM(CASE WHEN b.auction_id_64 IS NULL THEN
> 1 ELSE 0 END)
> FROM
>  data_subset_2 a
> LEFT OUTER JOIN
>  OrderFields b on a.auction_id_64=b.auction_id_64
> GROUP BY
>  a.clientid,  a.isi,  a.os,  a.bro,  a.engaged,  a.iframed,
>  hover, hower1, hower5, hower60,
>  Visible,  a.x1,  a.x2,  a.x3,
>  NoViewMeasure,  a.x4,  a.x5,  a.x6,
>  a.alerttype,    a.x7,  a.x9,  a.x11,  a.x13,  a.x17,  a.x21,  a.x27,  a.x28;
>
>
>
>
> 2012-05-18 17:49:32,440 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 17:50:32,813 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 17:51:33,176 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 17:52:33,538 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 17:53:33,906 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 17:54:34,270 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 17:55:34,644 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 17:56:35,017 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 17:57:35,387 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 17:58:35,754 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 17:59:36,125 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:00:36,503 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:01:36,873 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:02:36,990 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:03:37,355 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:04:37,719 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:05:38,081 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:06:38,432 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:07:38,777 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:08:39,135 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:09:39,489 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:10:39,852 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:11:40,221 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:12:40,591 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:13:40,962 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:14:41,336 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:15:41,700 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:16:42,070 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:17:42,440 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:18:42,801 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:19:43,156 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:20:43,531 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:21:43,899 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:22:44,255 Stage-1 map = 100%,  reduce = 97%
> 2012-05-18 18:23:44,619 Stage-1 map = 100%,  reduce = 97%
+
Balaji Rao 2012-05-20, 16:36
+
Abhishek Pratap Singh 2012-05-21, 21:23