Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> PIG 0.8.1 leaks Zookeeper connections when using HBaseStorage


Copy link to this message
-
Re: PIG 0.8.1 leaks Zookeeper connections when using HBaseStorage
Hey Vincent,

Will it be easy for you to isolate this in a test code. That will help to
debug the issue and also fixing it.

Ashutosh
On Fri, Aug 26, 2011 at 05:30, Vincent Barat <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I run PIG jobs from a Java process (using PigServer). Most of which use
> HBaseStorage to load data from HBase.
> Each job is run using a new PigServer object, and I correctly call
> pigServer.shutdown() when my pig server is no longer used.
>
> Nevertheless, after a few hours of run, I notice that the number of
> connections to my Zookeeper servers reach the limit (300 in my case).
> It appears that each job leak 4 or 5 Zookeeper connections.
>
> It was not the case with PIG 0.6.1
>
> To solve this issue (temporarily) by killing the process running PIG after
> a few set of jobs have been run : connections are correctly closed.
> My process don't use HBase by itself, only HBaseStorage, so I guess the
> leak is in the code of HBaseStorage: maybe to cnx to HBase are not closed.
>
> Should I report this a an issue ?
>
> Thanks for your help
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB