Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> EvalFunc finish() closing connections prematurely


+
Mike Sukmanowsky 2013-03-14, 21:05
+
Jonathan Coveney 2013-03-14, 21:24
+
Mike Sukmanowsky 2013-03-18, 20:53
+
Mike Sukmanowsky 2013-03-22, 15:28
+
Dmitriy Ryaboy 2013-03-26, 00:03
Copy link to this message
-
Re: EvalFunc finish() closing connections prematurely
Sorry about not getting to this, Mike!

Dmitry's idea is a good one. I still think that something is axing the
connections, possible between different map slots on the same mapper. I
would log very liberally and see what comes of it. And I'll try and carve
out some time.
2013/3/26 Dmitriy Ryaboy <[EMAIL PROTECTED]>

> Mike, have you tried adding logging to any EvalFunc methods that
> communicate with Mongo to see which of them is calling it after finish() ?
> Are you sure something else doesn't close Mongo connection for you?
>
>
> On Fri, Mar 22, 2013 at 8:28 AM, Mike Sukmanowsky <[EMAIL PROTECTED]>
> wrote:
>
> > Bump - any thoughts?
> >
> >
> > On Mon, Mar 18, 2013 at 4:53 PM, Mike Sukmanowsky <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Sure thing - here you go
> > > https://gist.github.com/msukmanowsky/198c878e1deaf4e470a9
> > >
> > > When initialized, the ParselyMetadataService creates a new Mongo and
> > Jedis
> > > instance which the EvalFunc queries using a public method fetch().
> > >  Instance of ParselyMetadataService also have a close() function which
> > > simply calls:
> > >
> > > this.mongoClient.close();
> > > this.redisClient.disconnect();
> > >
> > >
> > > On Thu, Mar 14, 2013 at 5:24 PM, Jonathan Coveney <[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> Can you perhaps share more of your implementation? I can imagine a
> > couple
> > >> of things which would cause errors like this. Are you making sure that
> > >> each
> > >> instance of EvalFunc is dealing with a different connection?
> > >>
> > >> That's what I'd take a look at first...if that isn't the issue, I can
> > look
> > >> into how final() is implemented.
> > >>
> > >>
> > >> 2013/3/14 Mike Sukmanowsky <[EMAIL PROTECTED]>
> > >>
> > >> > Hi there,
> > >> >
> > >> > I have an EvalFunc which uses an internal class that opens up
> > >> connections
> > >> > to a Redis and MongoDB server.  This class has a close() method
> which
> > >> > closes connections to both Redis and MongoDB.
> > >> >
> > >> > In my EvalFunc implementation, I placed the close() method
> > >> >
> > >> >     @Override
> > >> >   public void finish() {
> > >> >   meta.close();
> > >> >   }
> > >> >
> > >> > But this is leading to a
> > >> >
> > >> > java.lang.IllegalStateException: this Mongo has been closed
> > >> >
> > >> > indicating that Pig attempted to read from Mongo after a call to
> > >> finish().
> > >> >  Is this an error specific to running a Pig script locally?
> > >> >
> > >> > Mike
> > >> >
> > >> > --
> > >> > Mike Sukmanowsky
> > >> >
> > >> > Product Lead, http://parse.ly
> > >> > 989 Avenue of the Americas, 3rd Floor
> > >> > New York, NY  10018
> > >> > p: +1 (416) 953-4248
> > >> > e: [EMAIL PROTECTED]
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Mike Sukmanowsky
> > >
> > > Product Lead, http://parse.ly
> > > 989 Avenue of the Americas, 3rd Floor
> > > New York, NY  10018
> > > p: +1 (416) 953-4248
> > > e: [EMAIL PROTECTED]
> > >
> >
> >
> >
> > --
> > Mike Sukmanowsky
> >
> > Product Lead, http://parse.ly
> > 989 Avenue of the Americas, 3rd Floor
> > New York, NY  10018
> > p: +1 (416) 953-4248
> > e: [EMAIL PROTECTED]
> >
>