Hi,

I am getting an out of memory error while running Pig. I am running a pretty big job with one master node and over 100 worker nodes. Pig divides the execution in two map-reduce jobs. Both the jobs succeed with a small data set. With a large data set I get an out of memory error at the end of the first job. This happens right after the all the mappers and reducers of the first job are done and the second job hasn't started. Here is the error:

2012-06-18 03:15:29,565 [Low Memory Detector] INFO  org.apache.pig.impl.util.SpillableMemoryManager - first memory handler call - Collection threshold init = 187039744(182656K) used = 390873656(381712K) committed = 613744640(599360K) max = 699072512(682688K)
2012-06-18 03:15:31,137 [Low Memory Detector] INFO  org.apache.pig.impl.util.SpillableMemoryManager - first memory handler call- Usage threshold init = 187039744(182656K) used = 510001720(498048K) committed = 613744640(599360K) max = 699072512(682688K)
Exception in thread "IPC Client (47) connection to /10.217.23.253:9001 from hadoop" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.TaskReport.<init>(TaskReport.java:46)
at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
at org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:53)
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:236)
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:219)
at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:742)
Exception in thread "Low Memory Detector" java.lang.OutOfMemoryError: Java heap space
at sun.management.MemoryUsageCompositeData.getCompositeData(MemoryUsageCompositeData.java:40)
at sun.management.MemoryUsageCompositeData.toCompositeData(MemoryUsageCompositeData.java:34)
at sun.management.MemoryNotifInfoCompositeData.getCompositeData(MemoryNotifInfoCompositeData.java:42)
at sun.management.MemoryNotifInfoCompositeData.toCompositeData(MemoryNotifInfoCompositeData.java:36)
at sun.management.MemoryImpl.createNotification(MemoryImpl.java:168)
at sun.management.MemoryPoolImpl$CollectionSensor.triggerAction(MemoryPoolImpl.java:300)
at sun.management.Sensor.trigger(Sensor.java:120)

I will really appreciate and suggestions on how to go about debugging and rectifying this issue.

Thanks,
Pankaj

export HADOOP_HEAPSIZE=<something more than what it is right now>

Thanks,
Aniket

On Sun, Jun 17, 2012 at 11:16 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I am getting an out of memory error while running Pig. I am running a
> pretty big job with one master node and over 100 worker nodes. Pig divides
> the execution in two map-reduce jobs. Both the jobs succeed with a small
> data set. With a large data set I get an out of memory error at the end of
> the first job. This happens right after the all the mappers and reducers of
> the first job are done and the second job hasn't started. Here is the error:
>
> 2012-06-18 03:15:29,565 [Low Memory Detector] INFO
>  org.apache.pig.impl.util.SpillableMemoryManager - first memory handler
> call - Collection threshold init = 187039744(182656K) used > 390873656(381712K) committed = 613744640(599360K) max = 699072512(682688K)
> 2012-06-18 03:15:31,137 [Low Memory Detector] INFO
>  org.apache.pig.impl.util.SpillableMemoryManager - first memory handler
> call- Usage threshold init = 187039744(182656K) used = 510001720(498048K)
> committed = 613744640(599360K) max = 699072512(682688K)
> Exception in thread "IPC Client (47) connection to /10.217.23.253:9001from hadoop" java.lang.RuntimeException:
> java.lang.reflect.InvocationTargetException
> Caused by: java.lang.reflect.InvocationTargetException
> Caused by: java.lang.OutOfMemoryError: Java heap space
>        at org.apache.hadoop.mapred.TaskReport.<init>(TaskReport.java:46)
>        at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown
> Source)
>        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
>        at
> org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:53)
>        at
> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:236)
>        at
> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
>        at
> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:219)
>        at
> org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
>        at
> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807)
>        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:742)
> Exception in thread "Low Memory Detector" java.lang.OutOfMemoryError: Java
> heap space
>        at
> sun.management.MemoryUsageCompositeData.getCompositeData(MemoryUsageCompositeData.java:40)
>        at
> sun.management.MemoryUsageCompositeData.toCompositeData(MemoryUsageCompositeData.java:34)
>        at
> sun.management.MemoryNotifInfoCompositeData.getCompositeData(MemoryNotifInfoCompositeData.java:42)
>        at
> sun.management.MemoryNotifInfoCompositeData.toCompositeData(MemoryNotifInfoCompositeData.java:36)
>        at sun.management.MemoryImpl.createNotification(MemoryImpl.java:168)
>        at
> sun.management.MemoryPoolImpl$CollectionSensor.triggerAction(MemoryPoolImpl.java:300)
>        at sun.management.Sensor.trigger(Sensor.java:120)
>
> I will really appreciate and suggestions on how to go about debugging and
> rectifying this issue.
>
> Thanks,
> Pankaj
--
"...:::Aniket:::... Quetzalco@tl"

Awesome, that worked. Thanks.
To summarize the root cause and resolution as I understand it:
At the end of first job, PIG did a list on the job and apparently that shot its Heap size requirement up to 1.7G. The existing, default, heapsize for the PIG JVM didn't allow that and gave an out of memory error. Setting HADOOP_HEAPSIZE to 8192 set the max heap size of the PIG process to 8196m and the problem went away.

Thanks,
Pankaj
On Jun 18, 2012, at 1:23 AM, Aniket Mokashi wrote:

> export HADOOP_HEAPSIZE=<something more than what it is right now>
>
> Thanks,
> Aniket
>
> On Sun, Jun 17, 2012 at 11:16 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> I am getting an out of memory error while running Pig. I am running a
>> pretty big job with one master node and over 100 worker nodes. Pig divides
>> the execution in two map-reduce jobs. Both the jobs succeed with a small
>> data set. With a large data set I get an out of memory error at the end of
>> the first job. This happens right after the all the mappers and reducers of
>> the first job are done and the second job hasn't started. Here is the error:
>>
>> 2012-06-18 03:15:29,565 [Low Memory Detector] INFO
>> org.apache.pig.impl.util.SpillableMemoryManager - first memory handler
>> call - Collection threshold init = 187039744(182656K) used >> 390873656(381712K) committed = 613744640(599360K) max = 699072512(682688K)
>> 2012-06-18 03:15:31,137 [Low Memory Detector] INFO
>> org.apache.pig.impl.util.SpillableMemoryManager - first memory handler
>> call- Usage threshold init = 187039744(182656K) used = 510001720(498048K)
>> committed = 613744640(599360K) max = 699072512(682688K)
>> Exception in thread "IPC Client (47) connection to /10.217.23.253:9001from hadoop" java.lang.RuntimeException:
>> java.lang.reflect.InvocationTargetException
>> Caused by: java.lang.reflect.InvocationTargetException
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>       at org.apache.hadoop.mapred.TaskReport.<init>(TaskReport.java:46)
>>       at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown
>> Source)
>>       at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>       at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>       at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
>>       at
>> org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:53)
>>       at
>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:236)
>>       at
>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
>>       at
>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:219)
>>       at
>> org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
>>       at
>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807)
>>       at org.apache.hadoop.ipc.Client$Connection.run(Client.java:742)
>> Exception in thread "Low Memory Detector" java.lang.OutOfMemoryError: Java
>> heap space
>>       at
>> sun.management.MemoryUsageCompositeData.getCompositeData(MemoryUsageCompositeData.java:40)
>>       at
>> sun.management.MemoryUsageCompositeData.toCompositeData(MemoryUsageCompositeData.java:34)
>>       at
>> sun.management.MemoryNotifInfoCompositeData.getCompositeData(MemoryNotifInfoCompositeData.java:42)
>>       at
>> sun.management.MemoryNotifInfoCompositeData.toCompositeData(MemoryNotifInfoCompositeData.java:36)
>>       at sun.management.MemoryImpl.createNotification(MemoryImpl.java:168)
>>       at
>> sun.management.MemoryPoolImpl$CollectionSensor.triggerAction(MemoryPoolImpl.java:300)
>>       at sun.management.Sensor.trigger(Sensor.java:120)
>>
>> I will really appreciate and suggestions on how to go about debugging and
>> rectifying this issue.
>>
>> Thanks,
>> Pankaj
>
>
>
>
> --
> "...:::Aniket:::... Quetzalco@tl"

Kind of curious what it's using all that RAM for. Don't suppose you profiled it?

On Mon, Jun 18, 2012 at 1:37 PM, Pankaj Gupta <[EMAIL PROTECTED]> wrote:
> Awesome, that worked. Thanks.
> To summarize the root cause and resolution as I understand it:
> At the end of first job, PIG did a list on the job and apparently that shot its Heap size requirement up to 1.7G. The existing, default, heapsize for the PIG JVM didn't allow that and gave an out of memory error. Setting HADOOP_HEAPSIZE to 8192 set the max heap size of the PIG process to 8196m and the problem went away.
>
> Thanks,
> Pankaj
> On Jun 18, 2012, at 1:23 AM, Aniket Mokashi wrote:
>
>> export HADOOP_HEAPSIZE=<something more than what it is right now>
>>
>> Thanks,
>> Aniket
>>
>> On Sun, Jun 17, 2012 at 11:16 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:
>>
>>> Hi,
>>>
>>> I am getting an out of memory error while running Pig. I am running a
>>> pretty big job with one master node and over 100 worker nodes. Pig divides
>>> the execution in two map-reduce jobs. Both the jobs succeed with a small
>>> data set. With a large data set I get an out of memory error at the end of
>>> the first job. This happens right after the all the mappers and reducers of
>>> the first job are done and the second job hasn't started. Here is the error:
>>>
>>> 2012-06-18 03:15:29,565 [Low Memory Detector] INFO
>>> org.apache.pig.impl.util.SpillableMemoryManager - first memory handler
>>> call - Collection threshold init = 187039744(182656K) used >>> 390873656(381712K) committed = 613744640(599360K) max = 699072512(682688K)
>>> 2012-06-18 03:15:31,137 [Low Memory Detector] INFO
>>> org.apache.pig.impl.util.SpillableMemoryManager - first memory handler
>>> call- Usage threshold init = 187039744(182656K) used = 510001720(498048K)
>>> committed = 613744640(599360K) max = 699072512(682688K)
>>> Exception in thread "IPC Client (47) connection to /10.217.23.253:9001from hadoop" java.lang.RuntimeException:
>>> java.lang.reflect.InvocationTargetException
>>> Caused by: java.lang.reflect.InvocationTargetException
>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>       at org.apache.hadoop.mapred.TaskReport.<init>(TaskReport.java:46)
>>>       at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown
>>> Source)
>>>       at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>       at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>       at
>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
>>>       at
>>> org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:53)
>>>       at
>>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:236)
>>>       at
>>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
>>>       at
>>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:219)
>>>       at
>>> org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
>>>       at
>>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807)
>>>       at org.apache.hadoop.ipc.Client$Connection.run(Client.java:742)
>>> Exception in thread "Low Memory Detector" java.lang.OutOfMemoryError: Java
>>> heap space
>>>       at
>>> sun.management.MemoryUsageCompositeData.getCompositeData(MemoryUsageCompositeData.java:40)
>>>       at
>>> sun.management.MemoryUsageCompositeData.toCompositeData(MemoryUsageCompositeData.java:34)
>>>       at
>>> sun.management.MemoryNotifInfoCompositeData.getCompositeData(MemoryNotifInfoCompositeData.java:42)
>>>       at
>>> sun.management.MemoryNotifInfoCompositeData.toCompositeData(MemoryNotifInfoCompositeData.java:36)
>>>       at sun.management.MemoryImpl.createNotification(MemoryImpl.java:168)
>>>       at
>>> sun.management.MemoryPoolImpl$CollectionSensor.triggerAction(MemoryPoolImpl.java:300)
>>>       at sun.management.Sensor.trigger(Sensor.java:120)
>>>
>>> I will really appreciate and suggestions on how to go about debugging and

No, I didn't profile it. I am pretty sure this will happen again next time I run the job, so I can monitor it at that time, but I don't know enough about Pig at this point to be able to find out what it is doing with all that RAM. If you have any suggestions do let me know and I will try to check that next time I run this job.

Thanks,
Pankaj
On Jun 18, 2012, at 4:23 PM, Dmitriy Ryaboy wrote:

> Kind of curious what it's using all that RAM for. Don't suppose you profiled it?
>
> On Mon, Jun 18, 2012 at 1:37 PM, Pankaj Gupta <[EMAIL PROTECTED]> wrote:
>> Awesome, that worked. Thanks.
>> To summarize the root cause and resolution as I understand it:
>> At the end of first job, PIG did a list on the job and apparently that shot its Heap size requirement up to 1.7G. The existing, default, heapsize for the PIG JVM didn't allow that and gave an out of memory error. Setting HADOOP_HEAPSIZE to 8192 set the max heap size of the PIG process to 8196m and the problem went away.
>>
>> Thanks,
>> Pankaj
>> On Jun 18, 2012, at 1:23 AM, Aniket Mokashi wrote:
>>
>>> export HADOOP_HEAPSIZE=<something more than what it is right now>
>>>
>>> Thanks,
>>> Aniket
>>>
>>> On Sun, Jun 17, 2012 at 11:16 PM, Pankaj Gupta <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hi,
>>>>
>>>> I am getting an out of memory error while running Pig. I am running a
>>>> pretty big job with one master node and over 100 worker nodes. Pig divides
>>>> the execution in two map-reduce jobs. Both the jobs succeed with a small
>>>> data set. With a large data set I get an out of memory error at the end of
>>>> the first job. This happens right after the all the mappers and reducers of
>>>> the first job are done and the second job hasn't started. Here is the error:
>>>>
>>>> 2012-06-18 03:15:29,565 [Low Memory Detector] INFO
>>>> org.apache.pig.impl.util.SpillableMemoryManager - first memory handler
>>>> call - Collection threshold init = 187039744(182656K) used >>>> 390873656(381712K) committed = 613744640(599360K) max = 699072512(682688K)
>>>> 2012-06-18 03:15:31,137 [Low Memory Detector] INFO
>>>> org.apache.pig.impl.util.SpillableMemoryManager - first memory handler
>>>> call- Usage threshold init = 187039744(182656K) used = 510001720(498048K)
>>>> committed = 613744640(599360K) max = 699072512(682688K)
>>>> Exception in thread "IPC Client (47) connection to /10.217.23.253:9001from hadoop" java.lang.RuntimeException:
>>>> java.lang.reflect.InvocationTargetException
>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>       at org.apache.hadoop.mapred.TaskReport.<init>(TaskReport.java:46)
>>>>       at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown
>>>> Source)
>>>>       at
>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>>       at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>>       at
>>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
>>>>       at
>>>> org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:53)
>>>>       at
>>>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:236)
>>>>       at
>>>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
>>>>       at
>>>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:219)
>>>>       at
>>>> org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
>>>>       at
>>>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807)
>>>>       at org.apache.hadoop.ipc.Client$Connection.run(Client.java:742)
>>>> Exception in thread "Low Memory Detector" java.lang.OutOfMemoryError: Java
>>>> heap space
>>>>       at
>>>> sun.management.MemoryUsageCompositeData.getCompositeData(MemoryUsageCompositeData.java:40)
>>>>       at
>>>> sun.management.MemoryUsageCompositeData.toCompositeData(MemoryUsageCompositeData.java:34)
>>>>       at