Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> classloader question


+
Babak Farhang 2011-10-11, 23:39
+
Babak Farhang 2011-10-12, 02:39
+
Dmitriy Ryaboy 2011-10-12, 03:04
+
Babak Farhang 2011-10-12, 05:46
Copy link to this message
-
Re: classloader question
Take a look at the pig-withouthadoop target in the build.xml from your pig
release.  Usage of the target is documented here (for a different goal,
although):

http://thedatachef.blogspot.com/2011/01/apache-pig-08-with-cloudera-cdh3.html

Essentially, the target allows you to build pig without hadoop's JARs, but
then you're responsible for supplying the JARs back in at runtime (via
bin/pig modifications or some other method).

Norbert

On Wed, Oct 12, 2011 at 1:46 AM, Babak Farhang <[EMAIL PROTECTED]> wrote:

> Thanks for chiming in Dmitriy.
>
> Yes, I think I have verified that the version that ends up getting
> called in my udf is in fact the one bundled with hadoop; not the one I
> bundled w/ my udf jar. As I understand it, this is a common problem
> any "container" app such as hadoop (or say, tomcat) that loads
> "external" user-defined classes must deal with. Usually, the "how" of
> achieving the desired behavior involves setting up rules for which
> classes are looked up first in the parent loader (and if not found
> there then in the child loader) and which classes are first looked up
> in reverse (child loader first, then in parent loader). The rules that
> govern this behavior are called the "delegation model" for the [child]
> classloader. I'm new to this hadoop/pig environment and would like to
> learn what these rules are.
>
> Hope I've made my question clearer :-)
>
> Regards
> -Babak
>
> On Tue, Oct 11, 2011 at 8:04 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]>
> wrote:
> > I think the problem isn't the classloader -- it's the fact that both
> jackson
> > and joda are already bundled into the pig jar (presumably, different
> > versions of both libraries than the ones you are using). You need to
> either
> > repackage pig to not bundle those libraries, or make your code work with
> > Pig's versions of joda and jackson.
> >
> > D
> >
> > On Tue, Oct 11, 2011 at 7:39 PM, Babak Farhang <[EMAIL PROTECTED]>
> wrote:
> >
> >> Reading my original post over again, I see that I should have been
> >> clearer. I *am* including a copy of the versions of the jackson and
> >> joda libs that I need in my udf jar file. These libs are included in
> >> "exploded" form (i.e. not as embedded jars within the udf jar file,
> >> but in unzipped form alongside my own class files). However, they
> >> don't seem to get picked up by the hadoop/pig classloader. Am I doing
> >> this all wrong?
> >>
> >> On Tue, Oct 11, 2011 at 4:39 PM, Babak Farhang <[EMAIL PROTECTED]>
> wrote:
> >> > Greetings everyone,
> >> >
> >> > My pig script contains a call to my custom udf and I seem to be
> >> > running into a couple of classloader issues when running it. Below are
> >> > the specifics (the call stack), but I have some beginner general
> >> > questions regarding classloaders in pig:
> >> >
> >> > 1. Is there a way to configure the classloader used to load the udf
> >> > class and its deps? (I see, for example, a setClassloader method in
> >> > the impl PigContext class which is not directly exposed to the user)
> >> >
> >> > 2. What's delegation model does pig's udf classloader use when
> >> > resolving classes (e.g. parent classloader first, then child--or more
> >> > likely something a bit more complicated)?
> >> >
> >> > Any info/ideas you can share would be much appreciated.
> >> > Thx!
> >> >
> >> > -Babak
> >> >
> >> > ==============================> >> >
> >> > Now about the context/specifics of my error:
> >> >
> >> > My UDF uses the joda time and the jackson json libs (versions 2.0, and
> >> > 1.8.6, respectively) which I package along with my UDF in the jar that
> >> > ends up being registered in my pig script. Here are the call stacks:
> >> >
> >> > Joda-related:
> >> > Exception in thread "main" java.io.IOException: Resource not found:
> >> > "org/joda/time/tz/data/ZoneInfoMap" ClassLoader:
> >> > sun.misc.Launcher$AppClassLoader@4aad3ba4
> >> >        at
> >>
> org.joda.time.tz.ZoneInfoProvider.openResource(ZoneInfoProvider.java:211)
> >> >        at
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB