Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Adding dependent jars for UDF in the PIG


+
Darpan R 2013-08-13, 05:38
+
Ankit Bhatnagar 2013-08-13, 05:42
+
Serega Sheypak 2013-08-13, 05:58
+
Darpan R 2013-08-13, 08:30
+
Serega Sheypak 2013-08-13, 09:06
Copy link to this message
-
Re: Adding dependent jars for UDF in the PIG
Niels Basjes 2013-08-13, 10:10
I've had the exact same problem with my own udf and joda.
It turn out that an old version of joda is embedded inside the pig
application.
My workaround was limitying myself to the old joda api.

Niels
On Aug 13, 2013 10:31 AM, "Darpan R" <[EMAIL PROTECTED]> wrote:

> Thanks Serega and Ankit.
>
> According to my understanding it looks that the actual joda-time jar is in
> the classpath (otherwise it would have failed saying no classDef found).
> But it is not able to find the "now" static method which is part of the
> DateTime class.
> Not sure if I am missing something.
>
> The transformation I am doing on the date field using joda date time API.
>
> Let me paste the complete stack trace :
>
>
> 2013-08-11 13:01:06,911 [Thread-9] WARN
> org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
> java.lang.NoSuchMethodError:
>
> org.joda.time.DateTime.now(Lorg/joda/time/DateTimeZone;)Lorg/joda/time/DateTime;
>         at
> com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:178)
>         at
> com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:12)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:381)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:241)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:433)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
>         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>         at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:262)
> 2013-08-11 13:01:10,417 [main] WARN
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Ooops! Some job has failed! Specify -stop_on_fail
>
>
>
> Thanks again.
> DR
>
> On 13 August 2013 11:28, Serega Sheypak <[EMAIL PROTECTED]> wrote:
>
> > Pig uses reflection. The top exception says that there is no such method
> > signature. The problem is in the way you are trying to call method.
> > And its better to paste the whole stavktrace
> > 13.08.2013 7:39 пользователь "Darpan R" <[EMAIL PROTECTED]> написал:
> >
> > > I've a UDF which I use to do custom processing on the records. In the
> > eval
> > > function I am using a third party jar for processing. I saw the job jar
> > > file, but it does not include this dependency. Is there any way to
> > include
> > > dependent jar in the job jar ? (For testing I am running the cluster in
> > the
> > > local mode).
> > >
> > > Or can I use distributed cache to make the dependent jar available to
> the
> > > UDF ?
> > >
> > > I've tried registering the dependent jars in the pig. For the first
+
Serega Sheypak 2013-08-13, 10:15
+
Darpan R 2013-08-13, 10:31