Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> One file with sorted results.


+
sonia gehlot 2012-07-02, 23:59
+
Alan Gates 2012-07-03, 14:56
+
sonia gehlot 2012-07-03, 19:18
Copy link to this message
-
RE: One file with sorted results.
Have you tried breaking it into 2 jobs?  The first are the pre-sort work then a final job with the sort and single reducer?

Will Duckworth  Senior Vice President, Software Engineering  | comScore, Inc.(NASDAQ:SCOR)
o +1 (703) 438-2108 | m +1 (301) 606-2977 | mailto:[EMAIL PROTECTED]
.....................................................................................................

Introducing Mobile Metrix 2.0 - The next generation of mobile behavioral measurement
www.comscore.com/MobileMetrix
-----Original Message-----
From: sonia gehlot [mailto:[EMAIL PROTECTED]]
Sent: Monday, July 02, 2012 7:59 PM
To: [EMAIL PROTECTED]
Subject: One file with sorted results.

Hi Guys,

I have use case, where I need to generate data feed using Pig script. Data feed in total is of about 12 GB.

I want Pig script to generate 1 file and data in that data should be sorted as well. I know I can run it with one reducer as dataset is big with lot of Joins it takes forever to finish.

What are the other options to get one sorted file with better performance.

Thanks in advance,

Sonia