| clear query|facets|time |
Search criteria: .
Results from 31 to 40 from
60 (0.413s).
|
|
|
Loading phrases to help you
refine your search...
|
|
define global constants in pig script - Pig - [mail # user]
|
|
...Hi, all I got an idea about a new feature for pig: define global constants in pig script. Here's the example like this: define store_location "/where_you_store_result"; r...
|
|
|
Author: Haitao Yao,
2012-07-27, 03:32
|
|
|
[PIG-2827] TOP exception bug - Pig - [issue]
|
|
...the current exception will swallow the inner exception's stacktrace like this:java.lang.RuntimeException: General Exception executing function: java.lang.NullPointerException at org.apache.p...
|
|
|
http://issues.apache.org/jira/browse/PIG-2827
Author: Haitao Yao,
2012-07-19, 17:58
|
|
|
Re: What is the best way to do counting in pig? - Pig - [mail # user]
|
|
...Sorry , I sent the mail only to Thejas. Resend it for all. Haitao Yao [EMAIL PROTECTED] weibo: @haitao_yao Skype: haitao.yao.final 在 2012-7-12，上午10:41， Haitao Yao 写...
|
|
|
Author: Haitao Yao,
2012-07-12, 02:56
|
|
|
Re: What is the best way to do counting in pig? - Pig - [mail # user]
|
|
...Oh, new discovery: we can not set pig.cachedbag.memusage = 0 because every time the InternalCachedBag spills, It creates a new tmp file in java.io.tmpdir. if we set pig.cachedbag...
|
|
|
Author: Haitao Yao,
2012-07-11, 06:50
|
|
|
Re: pig reduce OOM - Pig - [mail # user]
|
|
...I've found the reason: it's InternalCachedBag. I've posted all the details in a mail titled: What is the best way to do counting in pig? I'm afraid I can not give you the mail link sin...
|
|
|
Author: Haitao Yao,
2012-07-10, 14:49
|
|
|
Re: How to CONCAT multiple expressions - Pig - [mail # user]
|
|
...like this: CONCAT(CONCAT('all', ':'), 'all') It's ugly , but the concat only accepts 2 parameters. Here's the java doc of CONCAT class: /** * Generates the con...
|
|
|
Author: Haitao Yao,
2012-07-10, 14:41
|
|
|
Re: What is the best way to do counting in pig? - Pig - [mail # user]
|
|
...I found the solution. After analyzing the heap dump while the reducer OOM, I found out the memory is consumed by org.apache.pig.data.InternalCachedBag , here's the diagram:...
|
|
|
Author: Haitao Yao,
2012-07-10, 08:20
|
|
|
Re: What is the best way to do counting in pig? - Pig - [mail # user]
|
|
...my reducers get 512 MB, -Xms512M -Xmx512M. The reducer does not get OOM when manually invoke spill in my case. Can you explain more about your solution? And can your solut...
|
|
|
Author: Haitao Yao,
2012-07-10, 05:06
|
|
|
Re: What is the best way to do counting in pig? - Pig - [mail # user]
|
|
...I have encountered the similar problem. And I got a OOM while running the reducer. I think the reason is the data bag generated after group all is too big to fit into the r...
|
|
|
Author: Haitao Yao,
2012-07-10, 02:28
|
|
|
Re: pig reduce OOM - Pig - [mail # user]
|
|
...seems like Big data big is still a headache for pig. here's a mail archive I found : http://mail-archives.apache.org/mod_mbox/pig-user/200806.mbox/%3C1b29507a0 [EMAIL PROTE...
|
|
|
Author: Haitao Yao,
2012-07-09, 10:24
|
|
|
|
