set -Xms low, say 64M. Then set -XX:MaxNewSize to about 100M. Then, it should grow from there to the default 1GB max more slowly, and you will be able to use your OS's process tools to see how much memory it really needs for each script.
The above means it will start with 100M heap, and grow it by some percentage whenever the total used space after a full GC is more than 60%. The -XX:MaxNewSize prevents the young generation from growing beyond that setting. By default the young generation is 1/3 of the heap, but for most applications and especially things like Pig, 100MB of young generation is plenty.
Also, if you are using a 64 bit JVM (Sun 6.0u14 or later) set -XX:+UseCompressedOops, this will save 25% or so space for many applications. On Jul 8, 2010, at 10:16 AM, Corbin Hoenes wrote:
> The default is 1 GB for pig to run. Has anyone had any success running with less say 512? We have a lot of jobs running at the same time--hoping to be able to run them with less ram. >
All projects made searchable here are trademarks of the Apache Software Foundation.
Service operated by Sematext