Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - classloader question


+
Babak Farhang 2011-10-11, 23:39
+
Babak Farhang 2011-10-12, 02:39
+
Dmitriy Ryaboy 2011-10-12, 03:04
+
Babak Farhang 2011-10-12, 05:46
Copy link to this message
-
Re: classloader question
Norbert Burger 2011-10-12, 12:34
Take a look at the pig-withouthadoop target in the build.xml from your pig
release.  Usage of the target is documented here (for a different goal,
although):

http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html

Essentially, the target allows you to build pig without hadoop's JARs, but
then you're responsible for supplying the JARs back in at runtime (via
bin/pig modifications or some other method).

Norbert

On Wed, Oct 12, 2011 at 1:46 AM, Babak Farhang <[EMAIL PROTECTED]> wrote:

> Thanks for chiming in Dmitriy.
>
> Yes, I think I have verified that the version that ends up getting
> called in my udf is in fact the one bundled with hadoop; not the one I
> bundled w/ my udf jar. As I understand it, this is a common problem
> any "container" app such as hadoop (or say, tomcat) that loads
> "external" user-defined classes must deal with. Usually, the "how" of
> achieving the desired behavior involves setting up rules for which
> classes are looked up first in the parent loader (and if not found
> there then in the child loader) and which classes are first looked up
> in reverse (child loader first, then in parent loader). The rules that
> govern this behavior are called the "delegation model" for the [child]
> classloader. I'm new to this hadoop/pig environment and would like to
> learn what these rules are.
>
> Hope I've made my question clearer :-)
>
> Regards
> -Babak
>
> On Tue, Oct 11, 2011 at 8:04 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]>
> wrote:
> > I think the problem isn't the classloader -- it's the fact that both
> jackson
> > and joda are already bundled into the pig jar (presumably, different
> > versions of both libraries than the ones you are using). You need to
> either
> > repackage pig to not bundle those libraries, or make your code work with
> > Pig's versions of joda and jackson.
> >
> > D
> >
> > On Tue, Oct 11, 2011 at 7:39 PM, Babak Farhang <[EMAIL PROTECTED]>
> wrote:
> >
> >> Reading my original post over again, I see that I should have been
> >> clearer. I *am* including a copy of the versions of the jackson and
> >> joda libs that I need in my udf jar file. These libs are included in
> >> "exploded" form (i.e. not as embedded jars within the udf jar file,
> >> but in unzipped form alongside my own class files). However, they
> >> don't seem to get picked up by the hadoop/pig classloader. Am I doing
> >> this all wrong?
> >>
> >> On Tue, Oct 11, 2011 at 4:39 PM, Babak Farhang <[EMAIL PROTECTED]>
> wrote:
> >> > Greetings everyone,
> >> >
> >> > My pig script contains a call to my custom udf and I seem to be
> >> > running into a couple of classloader issues when running it. Below are
> >> > the specifics (the call stack), but I have some beginner general
> >> > questions regarding classloaders in pig:
> >> >
> >> > 1. Is there a way to configure the classloader used to load the udf
> >> > class and its deps? (I see, for example, a setClassloader method in
> >> > the impl PigContext class which is not directly exposed to the user)
> >> >
> >> > 2. What's delegation model does pig's udf classloader use when
> >> > resolving classes (e.g. parent classloader first, then child--or more
> >> > likely something a bit more complicated)?
> >> >
> >> > Any info/ideas you can share would be much appreciated.
> >> > Thx!
> >> >
> >> > -Babak
> >> >
> >> > ==============================> >> >
> >> > Now about the context/specifics of my error:
> >> >
> >> > My UDF uses the joda time and the jackson json libs (versions 2.0, and
> >> > 1.8.6, respectively) which I package along with my UDF in the jar that
> >> > ends up being registered in my pig script. Here are the call stacks:
> >> >
> >> > Joda-related:
> >> > Exception in thread "main" java.io.IOException: Resource not found:
> >> > "org/joda/time/tz/data/ZoneInfoMap" ClassLoader:
> >> > sun.misc.Launcher$AppClassLoader@4aad3ba4
> >> >        at
> >>
> org.joda.time.tz.ZoneInfoProvider.openResource(ZoneInfoProvider.java:211)
> >> >        at