Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> PigServer memory leak


Copy link to this message
-
Re: PigServer memory leak
Great work, Bill in chasing this down. Can you open a jira putting all
this info + any additional info you have on this issue. If you can
also put script and steps to reproduce the memory leak, that will be
awesome.

Ashutosh

On Fri, Mar 19, 2010 at 10:34, Bill Graham <[EMAIL PROTECTED]> wrote:
> I believe I've found the cause of my Pig memory leak so I wanted to report
> back. I profiled my app after letting it run for a couple of days and found
> that the static toDelete Stack in the FileLocalizer object was growing over
> time without getting flushed. I had thousands of HFile objects in that
> stack. This produced a memory leak both in my app and in HDFS.
>
> The fix seems straightforward enough in my app. I suspect calling
> FileLocalizer.deleteTempFiles() after each usage of PigServer for a given
> execution of a given pig script will do the trick.
>
> This seems to be a major gotcha though that will likely burn others. I
> suggest we add FileLocalizer.deleteTempFiles() to the shutdown() method of
> PigServer. Thoughts?
>
> Currently shutdown isn't doing much:
>
>    public void shutdown() {
>        // clean-up activities
>            // TODO: reclaim scope to free up resources. Currently
>        // this is not implemented and throws an exception
>            // hence, for now, we won't call it.
>        //
>        // pigContext.getExecutionEngine().reclaimScope(this.scope);
>    }
>
> thanks,
> Bill
>
>
>
> On Wed, Mar 10, 2010 at 12:15 PM, Bill Graham <[EMAIL PROTECTED]> wrote:
>
>> Yes, these errors appear in the Pig client and the jobs are definitely
>> being executed on the cluster. I can see the data in HDFS and the jobs in
>> the JobTracker UI of the cluster.
>>
>>
>> On Wed, Mar 10, 2010 at 10:54 AM, Ashutosh Chauhan <
>> [EMAIL PROTECTED]> wrote:
>>
>>> [Low Memory Detector] [INFO] SpillableMemoryManager.java:143 low memory
>>> handler called
>>>
>>> Are you seeing this warning on client side, in pig logs? If so, then
>>> are you sure your job is actually running on real hadoop cluster.
>>> Because these logs should appear in task-tracker logs not in client
>>> logs. This may imply that you job is getting executed locally in local
>>> mode and not actually submitted to cluster. Look for the very first
>>> lines in the client logs, where Pig tries to connect to the cluster.
>>> See, if its successful in doing so.
>>>
>>>
>>>
>>> On Wed, Mar 10, 2010 at 10:15, Ashutosh Chauhan
>>> <[EMAIL PROTECTED]> wrote:
>>> > Posting for Bill.
>>> >
>>> >
>>> > ---------- Forwarded message ----------
>>> > From: Bill Graham <[EMAIL PROTECTED]>
>>> > Date: Wed, Mar 10, 2010 at 10:11
>>> > Subject: Re: PigServer memory leak
>>> > To: [EMAIL PROTECTED]
>>> >
>>> >
>>> > Thanks for the reply, Ashutosh.
>>> >
>>> > [hadoop.apache.org keeps flagging my reply as spam, so I'm replying
>>> > directly to you. Feel free to push this conversation back onto the
>>> > list, if you can. :)]
>>> >
>>> > I'm running the same two scripts, one after the other, every 5
>>> > minutes. The scripts have dynamic tokens substituted to change the
>>> > input and output directories. Besides that, they have the same logic.
>>> >
>>> > I will try to execute the script from grunt next time it happens, but
>>> > I don't see how a lack of pig MR optimizations could cause a memory
>>> > issue on the client? If I bounce my daemon, the next jobs to run
>>> > executes without a problem upon start, so I would also expect a script
>>> > run through grunt at that time to run without a problem as well.
>>> >
>>> > I reverted back to re-initializing PigServer for every run. I have
>>> > other places in my scheduled workflow where I interact with HDFS which
>>> > I've now modified to re-use an instance of Hadoop's Configuration
>>> > object for the life of the VM. I was re-initializing that many times
>>> > per run. Looking at the Configuration code it seems to re-parse the
>>> > XML configs into a DOM every time it's called, so this certainly looks