Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Classpath issues with a custom loadfunc


Copy link to this message
-
Classpath issues with a custom loadfunc
Niels Basjes 2013-04-16, 20:44
Hi,

I'm attempting to build a custom LoadFunc for pig and I'm running into a
rather silly issue.

My project has several dependencies and I've been trying to create a single
jar that contains all of these dependencies (I'm using the maven assembly
plugin).

Now if I manually place all dependencies in my pig script using a list of
"REGISTER foo.jar" statements it all works.
What I want is to have a single jar with everything in it so that I only
need a single jar in the pig code.

After some playing around I found that it I unpack all the dependencies
inside my jar and unpack them in the root of the jar it all seems to work
... and turns into a big mess with warnings about "[INFO]
META-INF/MANIFEST.MF already added, skipping".

What is the clean way of doing this and still let PIG find the dependencies
inside the jar?
Can anyone point me towards a working example?

Thanks.

--
Best regards / Met vriendelijke groeten,

Niels Basjes