Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Intermittent NullPointerException


Copy link to this message
-
Re: Intermittent NullPointerException
Hi Malcolm,

If you're not running in parallel, it may be a different issue. But I am
surprised that Pig 0.10 local mode fails Intermittently like you describe
w/o parallelism. You might have discovered a real issue. If you could
provide steps that reproduce the error, that would be great!

>> How do I tell which pig jar file I'm using currently ?

"pig -secretDebugCmd" will show which pig jar file in file system is picked
up. For example, it shows the following output for me:

/usr/bin/hadoop jar /home/cheolsoo/pig-svn/bin/../pig-withouthadoop.jar

Thanks,
Cheolsoo

On Mon, Nov 12, 2012 at 2:46 PM, Malcolm Tye <[EMAIL PROTECTED]>wrote:

> Hi Cheolsoo,
>                 I'm not specifically setting default_parallel in my script
> anywhere and I see this in the log file :-
>
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompi
> ler - Neither PARALLEL nor default parallelism is set for this job. Setting
> number of reducers to 1
>
> So I guess I'm not using parallel. Is it worth trying to compile Pig to use
> the Hadoop 0.23.x LocalJobRunner ? How do I tell which pig jar file I'm
> using currently ?
>
> Thanks
>
> Malc
>
>
> -----Original Message-----
> From: Cheolsoo Park [mailto:[EMAIL PROTECTED]]
> Sent: 12 November 2012 16:29
> To: [EMAIL PROTECTED]
> Subject: Re: Intermittent NullPointerException
>
> Hi Malcolm,
>
> How do you run your script? Do you run your script in parallel? Hadoop
> 1.0.x
> LocalJobRunner is not thread-safe, and Pig is by default built with Hadoop
> 1.0.x. I have seen a similar problem before (
> https://issues.apache.org/jira/browse/PIG-2852).
>
> If you're running your script in parallel, one workaround is to use Hdoop
> 0.23.x LocalJobRunner, which is thread-safe. You can do the following:
> - If you're using the standalone pig.jar, please download the Pig source
> tarball and run "ant clean jar -Dhadoopversion=23" to build pig.jar.
> - If you're using installed Hadoop with pig-withouthadoop.jar, please
> install Hadoop 0.23.x, download the Pig source tarball, and run "ant clean
> jar-withouthadoop -Dhadoopversion=23" to build pig-withouthadoop.jar.
>
> Hope this is helpful.
>
> Thanks,
> Cheolsoo
>
> On Mon, Nov 12, 2012 at 7:14 AM, Malcolm Tye
> <[EMAIL PROTECTED]>wrote:
>
> > Hi,****
> >
> >     I'm running Pig 0.10.0 in local mode on some small text files.
> > There is no intention to run it on Hadoop at all. We have a job that
> > runs every 5 minutes and about 3% of the time, the job fails with the
> > error below. It happens at random places within the Pig Script.****
> >
> > ** **
> >
> > 2012-10-19 14:15:37,719 [Thread-15] WARN
> > org.apache.hadoop.mapred.LocalJobRunner - job_local_0004
> > java.lang.NullPointerException
> >         at
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOp
> > erator.processInput(PhysicalOperator.java:286)
> >
> >         at
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expression
> > Operators.POProject.getNext(POProject.java:158)
> >
> >         at
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expression
> > Operators.POProject.getNext(POProject.java:360)
> >
> >         at
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOp
> > erator.getNext(PhysicalOperator.java:330)
> >
> >         at
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relational
> > Operators.POForEach.processPlan(POForEach.java:332)
> >
> >         at
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relational
> > Operators.POForEach.getNext(POForEach.java:284)
> >
> >         at
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOp
> > erator.processInput(PhysicalOperator.java:290)
> >
> >         at
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relational
> > Operators.POFilter.getNext(POFilter.java:95)
> >
> >         at
> > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOp
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB