Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Adding dependent jars for UDF in the PIG


Copy link to this message
-
Re: Adding dependent jars for UDF in the PIG
Serega Sheypak 2013-08-13, 09:06
The other version is:
You compiled tour UDF wih joda version X
But your runtime has joda version Y
2013/8/13 Darpan R <[EMAIL PROTECTED]>

> Thanks Serega and Ankit.
>
> According to my understanding it looks that the actual joda-time jar is in
> the classpath (otherwise it would have failed saying no classDef found).
> But it is not able to find the "now" static method which is part of the
> DateTime class.
> Not sure if I am missing something.
>
> The transformation I am doing on the date field using joda date time API.
>
> Let me paste the complete stack trace :
>
>
> 2013-08-11 13:01:06,911 [Thread-9] WARN
> org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
> java.lang.NoSuchMethodError:
>
> org.joda.time.DateTime.now(Lorg/joda/time/DateTimeZone;)Lorg/joda/time/DateTime;
>         at
> com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:178)
>         at
> com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:12)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:381)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:241)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:433)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
>         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>         at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:262)
> 2013-08-11 13:01:10,417 [main] WARN
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Ooops! Some job has failed! Specify -stop_on_fail
>
>
>
> Thanks again.
> DR
>
> On 13 August 2013 11:28, Serega Sheypak <[EMAIL PROTECTED]> wrote:
>
> > Pig uses reflection. The top exception says that there is no such method
> > signature. The problem is in the way you are trying to call method.
> > And its better to paste the whole stavktrace
> > 13.08.2013 7:39 пользователь "Darpan R" <[EMAIL PROTECTED]> написал:
> >
> > > I've a UDF which I use to do custom processing on the records. In the
> > eval
> > > function I am using a third party jar for processing. I saw the job jar
> > > file, but it does not include this dependency. Is there any way to
> > include
> > > dependent jar in the job jar ? (For testing I am running the cluster in
> > the
> > > local mode).
> > >
> > > Or can I use distributed cache to make the dependent jar available to
> the
> > > UDF ?
> > >
> > > I've tried registering the dependent jars in the pig. For the first
> > > registered jar (all udfs are bundled in this jar) I do not face the
> > issues.