Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Adding dependent jars for UDF in the PIG


Copy link to this message
-
Re: Adding dependent jars for UDF in the PIG
The other version is:
You compiled tour UDF wih joda version X
But your runtime has joda version Y
2013/8/13 Darpan R <[EMAIL PROTECTED]>

> Thanks Serega and Ankit.
>
> According to my understanding it looks that the actual joda-time jar is in
> the classpath (otherwise it would have failed saying no classDef found).
> But it is not able to find the "now" static method which is part of the
> DateTime class.
> Not sure if I am missing something.
>
> The transformation I am doing on the date field using joda date time API.
>
> Let me paste the complete stack trace :
>
>
> 2013-08-11 13:01:06,911 [Thread-9] WARN
> org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
> java.lang.NoSuchMethodError:
>
> org.joda.time.DateTime.now(Lorg/joda/time/DateTimeZone;)Lorg/joda/time/DateTime;
>         at
> com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:178)
>         at
> com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:12)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:381)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:241)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:433)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257)
>         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>         at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:262)
> 2013-08-11 13:01:10,417 [main] WARN
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Ooops! Some job has failed! Specify -stop_on_fail
>
>
>
> Thanks again.
> DR
>
> On 13 August 2013 11:28, Serega Sheypak <[EMAIL PROTECTED]> wrote:
>
> > Pig uses reflection. The top exception says that there is no such method
> > signature. The problem is in the way you are trying to call method.
> > And its better to paste the whole stavktrace
> > 13.08.2013 7:39 пользователь "Darpan R" <[EMAIL PROTECTED]> написал:
> >
> > > I've a UDF which I use to do custom processing on the records. In the
> > eval
> > > function I am using a third party jar for processing. I saw the job jar
> > > file, but it does not include this dependency. Is there any way to
> > include
> > > dependent jar in the job jar ? (For testing I am running the cluster in
> > the
> > > local mode).
> > >
> > > Or can I use distributed cache to make the dependent jar available to
> the
> > > UDF ?
> > >
> > > I've tried registering the dependent jars in the pig. For the first
> > > registered jar (all udfs are bundled in this jar) I do not face the
> > issues.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB