Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Java heap space error, for simple load/store on 1.5 gb file


Copy link to this message
-
Re: Java heap space error, for simple load/store on 1.5 gb file
Which version of pig are you using ?
Does your query have just a load and store statement ?
What load and store functions are you using ?
What would be the largest size of a single tuple in your data ?

Thanks,
Thejas
On 4/20/12 1:18 PM, shan s wrote:
> I am simply reading 1.5 gb file with 2458220 records, and storing it back.
>
> I am getting Java Heap space error
>
> The current setting is mapred.child.java.opts -Xmx1073741824
>
>
>
> Below is the error from log.
>
> 2012-04-20 15:03:15,769 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call-
> Usage threshold init = 350617600(342400K) used = 522736064(510484K)
> committed = 596443136(582464K) max = 715849728(699072K)
>
>
>
> Could you please help me interpret this.How do I set space for Java heap.
> I am looking at archive on this topic, but didn't see anything similar.
> (most of those talk about UDF, but I am simply doing a read/write).
>
> Thanks, Prashant.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB