Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Partition performance

Copy link to this message
Re: Partition performance
Owen O'Malley 2013-07-02, 14:51
On Tue, Jul 2, 2013 at 2:34 AM, Peter Marron <

>  Hi Owen,****
> ** **
> I’m curious about this advice about partitioning. Is there some
> fundamental reason why Hive****
> is slow when the number of partitions is 10,000 rather than 1,000?

The precise numbers don't matter. I wanted to give people a ballpark range
that they should be looking at. Most tables at 1000 partitions won't cause
big slow downs, but the cost scales with the number of partitions. By the
time you are at 10,000 the cost is noticeable. I have one customer who has
a table with 1.2 million partitions. That causes a lot of slow downs.
> And the improvements****
> that you mention are they going to be in version 12? Is there a JIRA
> raised so that I can track them?****
> (It’s not currently a problem for me but I can see that I am going to need
> to be able to explain the situation.)

I think this is the one they will use:

-- Owen