Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Intermittent NullPointerException


Copy link to this message
-
RE: Intermittent NullPointerException
Hi Cheolsoo,
The patch works as expected. We've not seen one error in the
test system since we installed the new jar file.

We're only processing ~200 rows at the most when we run the script, not sure
if that helps you narrow down the cause.

I assume we just use the patch you gave me on 0.10.0 until the fix comes out
in a later release ?

Many thanks for your quick response, it's very much appreciated.
Malc

-----Original Message-----
From: Cheolsoo Park [mailto:[EMAIL PROTECTED]]
Sent: 15 November 2012 00:16
To: [EMAIL PROTECTED]
Subject: Re: Intermittent NullPointerException

Hi Malcolm,

I have been running your script with 10M rows for a half day but couldn't
reproduce your error. So my analysis may be baseless here.

That being said, it looks like a race condition to me. The callstack in the
log shows below:

Caused by: java.lang.NullPointerException
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator
.processInput(PhysicalOperator.java:286)

Now if you look at PhysicalOperator.java:286, it's like this:

if(reporter!=null) {
    reporter.progress(); ---> NullPointerException is thrown here }

So 'reporter' became null between 'if(reporter!=null)' and
'reporter.progress()'.

Given that 'reporter' is a static field, this is totally possible.

public static PigProgressable reporter;

Even though you're setting default_parallel to 1, it only controls the
number of reducers, and the number of mappers is determined by the size of
input data. So you will still run multiple mapper threads in parallel in
LocalJobRunner, and they might be stepping into each other.

One possible fix is probably changing reporter to a thread local variable.
I will send a patch that does this to your email address. I based it to
branch-0.10, so you should be able to apply it cleanly to the 0.10 source
tarball running:

patch -p0 -i <patch file>

Can you please try to apply the patch, rebuild pig and see if that fixes
your problem? If this does, I will try to write a unit test case and commit
the fix upstream as well.

Thanks,
Cheolsoo

On Wed, Nov 14, 2012 at 4:32 AM, Malcolm Tye
<[EMAIL PROTECTED]>wrote:

> Hi,
>         Looks like zip files get rejected. Here's the log file
> unzipped
>
>
> Malc
>
>
> -----Original Message-----
> From: Malcolm Tye [mailto:[EMAIL PROTECTED]]
> Sent: 14 November 2012 12:01
> To: '[EMAIL PROTECTED]'
> Subject: RE: Intermittent NullPointerException
>
> Hi Cheolsoo,
>                 Even with the recompiled Pig, we still see the error.
> He's a debug log from Pig. It doesn't seem to give any more
> information.
>
> Any ideas ?
>
>
> Thanks
>
> Malc
>
>
> -----Original Message-----
> From: Malcolm Tye [mailto:[EMAIL PROTECTED]]
> Sent: 13 November 2012 12:58
> To: '[EMAIL PROTECTED]'
> Subject: RE: Intermittent NullPointerException
>
> Hi Cheolsoo,
>                 I tried setting default_parallel to 1 to rule out
> parallel processing, but the problem still happened.
>
> I've recompiled Pig and have put that into the test environment with
> the debug option set.
>
> I don't have recreate steps that fail every time. When the problem
> occurs, we can run the same script again on the input file  and the
> file gets processed OK the next time !
>
> Thanks
>
> Malc
>
>
> -----Original Message-----
> From: Cheolsoo Park [mailto:[EMAIL PROTECTED]]
> Sent: 12 November 2012 23:00
> To: [EMAIL PROTECTED]
> Subject: Re: Intermittent NullPointerException
>
> Hi Malcolm,
>
> If you're not running in parallel, it may be a different issue. But I
> am surprised that Pig 0.10 local mode fails Intermittently like you
> describe w/o parallelism. You might have discovered a real issue. If
> you could provide steps that reproduce the error, that would be great!
>
> >> How do I tell which pig jar file I'm using currently ?
>
> "pig -secretDebugCmd" will show which pig jar file in file system is
> picked up. For example, it shows the following output for me: