Hi,

I run PIG jobs from a Java process (using PigServer). Most of which
use HBaseStorage to load data from HBase.
Each job is run using a new PigServer object, and I correctly call
pigServer.shutdown() when my pig server is no longer used.

Nevertheless, after a few hours of run, I notice that the number of
connections to my Zookeeper servers reach the limit (300 in my case).
It appears that each job leak 4 or 5 Zookeeper connections.

It was not the case with PIG 0.6.1

To solve this issue (temporarily) by killing the process running PIG
after a few set of jobs have been run : connections are correctly
closed.
My process don't use HBase by itself, only HBaseStorage, so I guess
the leak is in the code of HBaseStorage: maybe to cnx to HBase are
not closed.

Should I report this a an issue ?

Thanks for your help

>
> Should I report this a an issue ?
>

Yes, please. I've found other resource leaks when using PigServer this way,
so this seems like a likely bug. Also, seeing that HTables are never closed
by HBaseStorage is not a good sign.
On Fri, Aug 26, 2011 at 5:30 AM, Vincent Barat <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I run PIG jobs from a Java process (using PigServer). Most of which use
> HBaseStorage to load data from HBase.
> Each job is run using a new PigServer object, and I correctly call
> pigServer.shutdown() when my pig server is no longer used.
>
> Nevertheless, after a few hours of run, I notice that the number of
> connections to my Zookeeper servers reach the limit (300 in my case).
> It appears that each job leak 4 or 5 Zookeeper connections.
>
> It was not the case with PIG 0.6.1
>
> To solve this issue (temporarily) by killing the process running PIG after
> a few set of jobs have been run : connections are correctly closed.
> My process don't use HBase by itself, only HBaseStorage, so I guess the
> leak is in the code of HBaseStorage: maybe to cnx to HBase are not closed.
>
> Should I report this a an issue ?
>
> Thanks for your help
>

Hey Vincent,

Will it be easy for you to isolate this in a test code. That will help to
debug the issue and also fixing it.

Ashutosh
On Fri, Aug 26, 2011 at 05:30, Vincent Barat <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I run PIG jobs from a Java process (using PigServer). Most of which use
> HBaseStorage to load data from HBase.
> Each job is run using a new PigServer object, and I correctly call
> pigServer.shutdown() when my pig server is no longer used.
>
> Nevertheless, after a few hours of run, I notice that the number of
> connections to my Zookeeper servers reach the limit (300 in my case).
> It appears that each job leak 4 or 5 Zookeeper connections.
>
> It was not the case with PIG 0.6.1
>
> To solve this issue (temporarily) by killing the process running PIG after
> a few set of jobs have been run : connections are correctly closed.
> My process don't use HBase by itself, only HBaseStorage, so I guess the
> leak is in the code of HBaseStorage: maybe to cnx to HBase are not closed.
>
> Should I report this a an issue ?
>
> Thanks for your help
>

I'm affraid it will be difficult. Nevertheles, I have cut & paste
some of the code used.

The issue is https://issues.apache.org/jira/browse/PIG-2251

By the way, I use PIG 0.8.1 +
https://issues.apache.org/jira/browse/PIG-2193

Le 26/08/11 18:19, Ashutosh Chauhan a �crit :
> Hey Vincent,
>
> Will it be easy for you to isolate this in a test code. That will help to
> debug the issue and also fixing it.
>
> Ashutosh
> On Fri, Aug 26, 2011 at 05:30, Vincent Barat<[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> I run PIG jobs from a Java process (using PigServer). Most of which use
>> HBaseStorage to load data from HBase.
>> Each job is run using a new PigServer object, and I correctly call
>> pigServer.shutdown() when my pig server is no longer used.
>>
>> Nevertheless, after a few hours of run, I notice that the number of
>> connections to my Zookeeper servers reach the limit (300 in my case).
>> It appears that each job leak 4 or 5 Zookeeper connections.
>>
>> It was not the case with PIG 0.6.1
>>
>> To solve this issue (temporarily) by killing the process running PIG after
>> a few set of jobs have been run : connections are correctly closed.
>> My process don't use HBase by itself, only HBaseStorage, so I guess the
>> leak is in the code of HBaseStorage: maybe to cnx to HBase are not closed.
>>
>> Should I report this a an issue ?
>>
>> Thanks for your help
>>