Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - UDFContext NULL JobConf


+
Henning Kropp 2013-10-30, 16:57
+
Cheolsoo Park 2013-10-30, 17:36
+
Henning Kropp 2013-10-30, 17:46
+
Pradeep Gollakota 2013-10-30, 17:58
+
Henning Kropp 2013-10-30, 18:34
Copy link to this message
-
Re: UDFContext NULL JobConf
Cheolsoo Park 2013-10-30, 19:09
Indeed. Do you mind filing a jira?
https://issues.apache.org/jira/browse/PIG
On Wed, Oct 30, 2013 at 11:34 AM, Henning Kropp <[EMAIL PROTECTED]>wrote:

> Wow, I was pretty sure to have tested it, but apparently not. The reason
> for the NPE seems to be the @MonitoredUDF
>
> You can reproduced it with the following code:
>
> import java.io.IOException;
> import java.util.concurrent.TimeUnit;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.pig.EvalFunc;
> import org.apache.pig.builtin.MonitoredUDF;
> import org.apache.pig.data.Tuple;
> import org.apache.pig.impl.util.UDFContext;
>
> @MonitoredUDF(timeUnit = TimeUnit.HOURS, duration = 3, intDefault = 1)
> public class TestJobConf extends EvalFunc<Tuple> {
>
>     public TestJobConf() {
>     }
>
>     @Override
>     public Tuple exec(Tuple tuple) throws IOException {
>         if (tuple == null || tuple.size() == 0) {
>             return null;
>         }
>         Configuration jobConf;
>         jobConf = UDFContext.getUDFContext().getJobConf();
>         System.err.println(jobConf.toString());
>         return null;
>     }
>
> }
>
> Log from the Task:
>
> 2013-10-30 19:24:09,516 ERROR TestJobConf:
> java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> 2013-10-30 19:24:10,388 ERROR TestJobConf:
> java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> 2013-10-30 19:24:10,880 ERROR TestJobConf:
> java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> 2013-10-30 19:24:11,512 ERROR TestJobConf:
> java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> 2013-10-30 19:24:12,264 ERROR TestJobConf:
> java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
>
>
> Without the MonitoredUDF Annotation the log output is:
>
> Configuration: core-default.xml, core-site.xml, mapred-default.xml,
> mapred-site.xml, hdfs-default.xml, hdfs-site.xml, .....
>
> Seems like a bug.
>
>
> 2013/10/30 Pradeep Gollakota <[EMAIL PROTECTED]>
>
> > Are you able to post your UDF (or at least a sanitized version)?
> >
> >
> > On Wed, Oct 30, 2013 at 10:46 AM, Henning Kropp <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi,
> > >
> > > thanks for your reply. I read about the expected behavior on the
> > front-end
> > > and I am getting the NPE on the back-end. The Mappers log the Exception
> > > during Execution.
> > >
> > > I am currently digging through debug messages. What to look out for?
> > There
> > > are bunch of
> > >
> > > [main] DEBUG org.apache.hadoop.conf.Configuration  -
> java.io.IOException:
> > > config()
> > >
> > > log messages. But I recall them as being "normal" for reasons I don't
> > > remember.
> > >
> > > Regards
> > >
> > >
> > > 2013/10/30 Cheolsoo Park <[EMAIL PROTECTED]>
> > >
> > > > Hi,
> > > >
> > > > Are you getting NPE on the front-end or the back-end? Sounds like
> > jobConf
> > > > is not added to UDFContext, which is expected on the front-end.
> Please
> > > see
> > > > the comments in getJobConf() and addJobConf() in the source code:
> > > >
> > > >
> > > >
> > >
> >
> https://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/UDFContext.java
> > > >
> > > > Thanks,
> > > > Cheolsoo
> > > >
> > > >
> > > > On Wed, Oct 30, 2013 at 9:57 AM, Henning Kropp <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am stuck. In my UDF (Java) extends EvalFunc the following code
> > throws
> > > > and
> > > > > NPE in exec(), when executed in -x mapred mode:
> > > > >
> > > > > Configuration jobConf = UDFContext.getUDFContext().getJobConf();
> > > > > System.err.println(jobConf.toString());
> > > > >
> > > > > I did not find any useful information as why my JobConf is always
> > null.
> > > > All
> > > > > I find is that this is the right way to get the JobConf in a UDF
> and
> > > that
> > > > > the behavior of what is returned when running locally (jira issue).
> > > > >
> > > > > Any ideas? I am running it on a very old Hadoop version 0.20.2 Are
+
Henning Kropp 2013-10-31, 07:49
+
Henning Kropp 2013-10-30, 16:43