Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Adding dependent jars for UDF in the PIG


Copy link to this message
-
Re: Adding dependent jars for UDF in the PIG
If you run your script using oozie, you can control it using
 <property>
                    <name>mapreduce.task.classpath.user.precedence</name>
                    <value>true</value>
                </property>

Also try to add to add to the top of your script:
set mapreduce.task.classpath.user.precedence=true
2013/8/13 Niels Basjes <[EMAIL PROTECTED]>

> I've had the exact same problem with my own udf and joda.
> It turn out that an old version of joda is embedded inside the pig
> application.
> My workaround was limitying myself to the old joda api.
>
> Niels
> On Aug 13, 2013 10:31 AM, "Darpan R" <[EMAIL PROTECTED]> wrote:
>
> > Thanks Serega and Ankit.
> >
> > According to my understanding it looks that the actual joda-time jar is
> in
> > the classpath (otherwise it would have failed saying no classDef found).
> > But it is not able to find the "now" static method which is part of the
> > DateTime class.
> > Not sure if I am missing something.
> >
> > The transformation I am doing on the date field using joda date time API.
> >
> > Let me paste the complete stack trace :
> >
> >
> > 2013-08-11 13:01:06,911 [Thread-9] WARN
> > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
> > java.lang.NoSuchMethodError:
> >
> >
> org.joda.time.DateTime.now(Lorg/joda/time/DateTimeZone;)Lorg/joda/time/DateTime;
> >         at
> >
> com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:178)
> >         at
> > com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:12)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:381)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:241)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:433)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
> >         at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
> >         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> >         at
> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650)
> >         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> >         at
> > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:262)
> > 2013-08-11 13:01:10,417 [main] WARN
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - Ooops! Some job has failed! Specify -stop_on_fail
> >
> >
> >
> > Thanks again.
> > DR
> >
> > On 13 August 2013 11:28, Serega Sheypak <[EMAIL PROTECTED]>
> wrote:
> >
> > > Pig uses reflection. The top exception says that there is no such
> method
> > > signature. The problem is in the way you are trying to call method.
> > > And its better to paste the whole stavktrace
> > > 13.08.2013 7:39 пользователь "Darpan R" <[EMAIL PROTECTED]> написал: