Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> PIG 0.8.1 leaks Zookeeper connections when using HBaseStorage


+
Vincent Barat 2011-08-26, 12:30
+
Bill Graham 2011-08-26, 16:17
Copy link to this message
-
Re: PIG 0.8.1 leaks Zookeeper connections when using HBaseStorage
Hey Vincent,

Will it be easy for you to isolate this in a test code. That will help to
debug the issue and also fixing it.

Ashutosh
On Fri, Aug 26, 2011 at 05:30, Vincent Barat <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I run PIG jobs from a Java process (using PigServer). Most of which use
> HBaseStorage to load data from HBase.
> Each job is run using a new PigServer object, and I correctly call
> pigServer.shutdown() when my pig server is no longer used.
>
> Nevertheless, after a few hours of run, I notice that the number of
> connections to my Zookeeper servers reach the limit (300 in my case).
> It appears that each job leak 4 or 5 Zookeeper connections.
>
> It was not the case with PIG 0.6.1
>
> To solve this issue (temporarily) by killing the process running PIG after
> a few set of jobs have been run : connections are correctly closed.
> My process don't use HBase by itself, only HBaseStorage, so I guess the
> leak is in the code of HBaseStorage: maybe to cnx to HBase are not closed.
>
> Should I report this a an issue ?
>
> Thanks for your help
>
+
Vincent Barat 2011-08-29, 13:43