Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Java heap space error, for simple load/store on 1.5 gb file


Copy link to this message
-
Re: Java heap space error, for simple load/store on 1.5 gb file
Thejas Nair 2012-04-21, 03:25
Which version of pig are you using ?
Does your query have just a load and store statement ?
What load and store functions are you using ?
What would be the largest size of a single tuple in your data ?

Thanks,
Thejas
On 4/20/12 1:18 PM, shan s wrote:
> I am simply reading 1.5 gb file with 2458220 records, and storing it back.
>
> I am getting Java Heap space error
>
> The current setting is mapred.child.java.opts -Xmx1073741824
>
>
>
> Below is the error from log.
>
> 2012-04-20 15:03:15,769 INFO
> org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call-
> Usage threshold init = 350617600(342400K) used = 522736064(510484K)
> committed = 596443136(582464K) max = 715849728(699072K)
>
>
>
> Could you please help me interpret this.How do I set space for Java heap.
> I am looking at archive on this topic, but didn't see anything similar.
> (most of those talk about UDF, but I am simply doing a read/write).
>
> Thanks, Prashant.
>