| clear query|facets|time |
Search criteria: .
Results from 21 to 30 from
64 (0.098s).
|
|
|
Loading phrases to help you
refine your search...
|
|
Re: pig script takes much longer than java MR job - Pig - [mail # user]
|
|
...Yeah sounds like a lot to dump if it takes 15 minutes to run. That alone can take long time. I once forgot to comment out some debug line in my udf. When run with produc t...
|
|
|
Author: Dexin Wang,
2011-06-18, 00:34
|
|
|
Re: running pig on amazon ec2 - Pig - [mail # user]
|
|
...Thanks a lot for the good advice. I'll see if I can get lzo setup. Currently I'm using emr which uses pig 0.6. I'll looking into whirr to start the hadoop cluster on ec2. There i...
|
|
|
Author: Dexin Wang,
2011-06-16, 04:16
|
|
|
Re: running pig on amazon ec2 - Pig - [mail # user]
|
|
...Tomas, What worked well for me is still to be figured out. Right now, it works but it's too slow. I think one of the main problem is that my job has many JOIN/GROUP BY, so lots of inte...
|
|
|
Author: Dexin Wang,
2011-06-15, 18:14
|
|
|
Re: running pig on amazon ec2 - Pig - [mail # user]
|
|
...Good to know. Trying single node hadoop cluster now. The main input is about 1+ million lines of events. After some aggregation, it joins with another input source which has also about 1+ mi...
|
|
|
Author: Dexin Wang,
2011-06-14, 18:07
|
|
|
Re: running pig on amazon ec2 - Pig - [mail # user]
|
|
...Thanks for your feedback. My comments below. On Tue, Jun 14, 2011 at 10:41 AM, Daniel Dai wrote: Local mode (-x local) when I ran it on my laptop, and mapreduce mode when I...
|
|
|
Author: Dexin Wang,
2011-06-14, 17:54
|
|
|
running pig on amazon ec2 - Pig - [mail # user]
|
|
...Hi, This is probably not directly a Pig question. Anyone running Pig on amazon EC2 instances? Something's not making sense to me. I ran a Pig script that has about 10 mapred jobs...
|
|
|
Author: Dexin Wang,
2011-06-13, 18:54
|
|
|
Re: Setting the store file name with date - Pig - [mail # user]
|
|
...I don't think version is a problem. variables is probably supported from th e start of the Pig. Using STORE result INTO 'out-$date'; I mentioned about, when y...
|
|
|
Author: Dexin Wang,
2011-05-23, 16:18
|
|
|
Re: Setting the store file name with date - Pig - [mail # user]
|
|
...Yeah I do that all the time. STORE result INTO 'out-$date'; Or you could run the pig script then after it's done move the result aside. On May 20, 2011, at 6:51 PM...
|
|
|
Author: Dexin Wang,
2011-05-21, 02:44
|
|
|
Re: elephantbird JsonLoader doesn't like gz? - Pig - [mail # user]
|
|
...Turns out it's only a problem if I run it in local mode, running it in cluster doesn't have this problem. I'm using EB1.2.5. Wonder how you fix the problem since it seems it's not EB p...
|
|
|
Author: Dexin Wang,
2011-05-19, 04:32
|
|
|
Re: elephantbird JsonLoader doesn't like gz? - Pig - [mail # user]
|
|
...Or is it because I'm using Pig 0.6 where gz format is not supported? I'll run this on aws EMR which only pig 0.6 is supported. I have to use later version of Pig? On Wed, May 18, 2011 ...
|
|
|
Author: Dexin Wang,
2011-05-18, 18:26
|
|
|
|
