Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Hive Group By Limitations


Copy link to this message
-
Re: Hive Group By Limitations
Nitin Pawar 2013-05-06, 18:31
best way to do all this would be run a distinct and group by along side a
join (its just a  guess but a more detailed approach other guys will
suggest )
On Mon, May 6, 2013 at 11:57 PM, Peter Chu <[EMAIL PROTECTED]> wrote:

> Thanks Nitin and Michael,
>
> The reason I asked is because I cannot help but wonder if it takes extra
> time with all those group by columns.
>
> Say for example, I have a employees table with 10 columns pertaining to
> employees but there could be duplicates, I need to de dup it by performing
> a group by employee id, and hire date, but were I want to select the other
> fields as well, I would have to add those fields in the group by clause, or
> join it back in another table
>
> When the table is big, I cannot help but wonder if it hurts the
> performance.
>
> Peter
>
> ------------------------------
> Date: Mon, 6 May 2013 23:42:23 +0530
> Subject: Re: Hive Group By Limitations
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
>
> hi Peter,
>
> In hive if you are running a group by, then all the select columns have to
> be in the group by clause. This limitation is for the column definition
> only and not for the column operations like count etc
>
> All the columns for group by do go to a single map reduce job and it does
> not launch multiple mapreduce jobs for each group by.
>
> I am not sure what do you mean by better way?
>
>
>
>
> On Mon, May 6, 2013 at 11:37 PM, Peter Chu <[EMAIL PROTECTED]> wrote:
>
> In Hive, I cannot perform a SELECT GROUP BY on fields not in the GROUP BY
> clause.
>
> Example: SELECT st.a, st.b, st.c, st.d, FROM some_table st GROUP BY st.a;
>  -- This does not work.
>
> To make it work, I would need to add the other fields in the group by
> clause.
>
> Not quite sure but I think each group by will give another M/R job.
>
> Wondering if there is any other way / better way to do group by.
>
> Peter
>
>
>
>
> --
> Nitin Pawar
>

--
Nitin Pawar