Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Adding dependent jars for UDF in the PIG


Copy link to this message
-
Re: Adding dependent jars for UDF in the PIG
If you run your script using oozie, you can control it using
 <property>
                    <name>mapreduce.task.classpath.user.precedence</name>
                    <value>true</value>
                </property>

Also try to add to add to the top of your script:
set mapreduce.task.classpath.user.precedence=true
2013/8/13 Niels Basjes <[EMAIL PROTECTED]>

> I've had the exact same problem with my own udf and joda.
> It turn out that an old version of joda is embedded inside the pig
> application.
> My workaround was limitying myself to the old joda api.
>
> Niels
> On Aug 13, 2013 10:31 AM, "Darpan R" <[EMAIL PROTECTED]> wrote:
>
> > Thanks Serega and Ankit.
> >
> > According to my understanding it looks that the actual joda-time jar is
> in
> > the classpath (otherwise it would have failed saying no classDef found).
> > But it is not able to find the "now" static method which is part of the
> > DateTime class.
> > Not sure if I am missing something.
> >
> > The transformation I am doing on the date field using joda date time API.
> >
> > Let me paste the complete stack trace :
> >
> >
> > 2013-08-11 13:01:06,911 [Thread-9] WARN
> > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
> > java.lang.NoSuchMethodError:
> >
> >
> org.joda.time.DateTime.now(Lorg/joda/time/DateTimeZone;)Lorg/joda/time/DateTime;
> >         at
> >
> com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:178)
> >         at
> > com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:12)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:381)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:241)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:433)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
> >         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> >         at
> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650)
> >         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> >         at
> > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:262)
> > 2013-08-11 13:01:10,417 [main] WARN
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - Ooops! Some job has failed! Specify -stop_on_fail
> >
> >
> >
> > Thanks again.
> > DR
> >
> > On 13 August 2013 11:28, Serega Sheypak <[EMAIL PROTECTED]>
> wrote:
> >
> > > Pig uses reflection. The top exception says that there is no such
> method
> > > signature. The problem is in the way you are trying to call method.
> > > And its better to paste the whole stavktrace
> > > 13.08.2013 7:39 пользователь "Darpan R" <[EMAIL PROTECTED]> написал:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB