Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> UDFContext NULL JobConf


+
Henning Kropp 2013-10-30, 16:57
+
Cheolsoo Park 2013-10-30, 17:36
+
Henning Kropp 2013-10-30, 17:46
+
Pradeep Gollakota 2013-10-30, 17:58
+
Henning Kropp 2013-10-30, 18:34
Copy link to this message
-
Re: UDFContext NULL JobConf
Indeed. Do you mind filing a jira?
https://issues.apache.org/jira/browse/PIG
On Wed, Oct 30, 2013 at 11:34 AM, Henning Kropp <[EMAIL PROTECTED]>wrote:

> Wow, I was pretty sure to have tested it, but apparently not. The reason
> for the NPE seems to be the @MonitoredUDF
>
> You can reproduced it with the following code:
>
> import java.io.IOException;
> import java.util.concurrent.TimeUnit;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.pig.EvalFunc;
> import org.apache.pig.builtin.MonitoredUDF;
> import org.apache.pig.data.Tuple;
> import org.apache.pig.impl.util.UDFContext;
>
> @MonitoredUDF(timeUnit = TimeUnit.HOURS, duration = 3, intDefault = 1)
> public class TestJobConf extends EvalFunc<Tuple> {
>
>     public TestJobConf() {
>     }
>
>     @Override
>     public Tuple exec(Tuple tuple) throws IOException {
>         if (tuple == null || tuple.size() == 0) {
>             return null;
>         }
>         Configuration jobConf;
>         jobConf = UDFContext.getUDFContext().getJobConf();
>         System.err.println(jobConf.toString());
>         return null;
>     }
>
> }
>
> Log from the Task:
>
> 2013-10-30 19:24:09,516 ERROR TestJobConf:
> java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> 2013-10-30 19:24:10,388 ERROR TestJobConf:
> java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> 2013-10-30 19:24:10,880 ERROR TestJobConf:
> java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> 2013-10-30 19:24:11,512 ERROR TestJobConf:
> java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> 2013-10-30 19:24:12,264 ERROR TestJobConf:
> java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
>
>
> Without the MonitoredUDF Annotation the log output is:
>
> Configuration: core-default.xml, core-site.xml, mapred-default.xml,
> mapred-site.xml, hdfs-default.xml, hdfs-site.xml, .....
>
> Seems like a bug.
>
>
> 2013/10/30 Pradeep Gollakota <[EMAIL PROTECTED]>
>
> > Are you able to post your UDF (or at least a sanitized version)?
> >
> >
> > On Wed, Oct 30, 2013 at 10:46 AM, Henning Kropp <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi,
> > >
> > > thanks for your reply. I read about the expected behavior on the
> > front-end
> > > and I am getting the NPE on the back-end. The Mappers log the Exception
> > > during Execution.
> > >
> > > I am currently digging through debug messages. What to look out for?
> > There
> > > are bunch of
> > >
> > > [main] DEBUG org.apache.hadoop.conf.Configuration  -
> java.io.IOException:
> > > config()
> > >
> > > log messages. But I recall them as being "normal" for reasons I don't
> > > remember.
> > >
> > > Regards
> > >
> > >
> > > 2013/10/30 Cheolsoo Park <[EMAIL PROTECTED]>
> > >
> > > > Hi,
> > > >
> > > > Are you getting NPE on the front-end or the back-end? Sounds like
> > jobConf
> > > > is not added to UDFContext, which is expected on the front-end.
> Please
> > > see
> > > > the comments in getJobConf() and addJobConf() in the source code:
> > > >
> > > >
> > > >
> > >
> >
> https://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/UDFContext.java
> > > >
> > > > Thanks,
> > > > Cheolsoo
> > > >
> > > >
> > > > On Wed, Oct 30, 2013 at 9:57 AM, Henning Kropp <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am stuck. In my UDF (Java) extends EvalFunc the following code
> > throws
> > > > and
> > > > > NPE in exec(), when executed in -x mapred mode:
> > > > >
> > > > > Configuration jobConf = UDFContext.getUDFContext().getJobConf();
> > > > > System.err.println(jobConf.toString());
> > > > >
> > > > > I did not find any useful information as why my JobConf is always
> > null.
> > > > All
> > > > > I find is that this is the right way to get the JobConf in a UDF
> and
> > > that
> > > > > the behavior of what is returned when running locally (jira issue).
> > > > >
> > > > > Any ideas? I am running it on a very old Hadoop version 0.20.2 Are
+
Henning Kropp 2013-10-31, 07:49
+
Henning Kropp 2013-10-30, 16:43
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB