Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Classpath issues with a custom loadfunc


Copy link to this message
-
Classpath issues with a custom loadfunc
Hi,

I'm attempting to build a custom LoadFunc for pig and I'm running into a
rather silly issue.

My project has several dependencies and I've been trying to create a single
jar that contains all of these dependencies (I'm using the maven assembly
plugin).

Now if I manually place all dependencies in my pig script using a list of
"REGISTER foo.jar" statements it all works.
What I want is to have a single jar with everything in it so that I only
need a single jar in the pig code.

After some playing around I found that it I unpack all the dependencies
inside my jar and unpack them in the root of the jar it all seems to work
... and turns into a big mess with warnings about "[INFO]
META-INF/MANIFEST.MF already added, skipping".

What is the clean way of doing this and still let PIG find the dependencies
inside the jar?
Can anyone point me towards a working example?

Thanks.

--
Best regards / Met vriendelijke groeten,

Niels Basjes
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB