Avro, mail # user - run time error during reduce stage: No field named ____ in: null

run time error during reduce stage: No field named ____ in: null

Brian Derickson 2012-11-01, 22:09

I've been pulling my hair out over this all day, and I'm hoping this is
something simple I'm overlooking.

The relevant portions of my code, the schema I'm using, and the stack trace
are at https://gist.github.com/3996847.

I'm using Hadoop 0.20.2 and Avro 1.7.1 as part of CDH4.

To briefly describe what I'm doing: the mapper (not included in the gist)
is taking a bam file and spitting out some information. The key is the
chromosome and position colon delimited and the value is an integer.

The reducer is summing up all the integers at a particular position and
creating a Pair object containing a record using the schema included in my
gist. The second portion of the pair is an integer that I don't care
about... if I didn't use a Pair here, I'd get an error. If this is
something I could do differently, please correct me. :)

Every time this is run, I get the stack trace included in the gist. I've
run out of things to try to fix this... I'd really really appreciate any
help I can get. Thanks!

Re: run time error during reduce stage: No field named ____ in: null

Dave Beech 2012-11-01, 22:49

Hi Brian

I don't think the output from the reducer should be a Pair. You said
you got an error when you didn't use a Pair here - what was it?

Cheers,
Dave

On 1 November 2012 22:09, Brian Derickson <[EMAIL PROTECTED]> wrote:
> I've been pulling my hair out over this all day, and I'm hoping this is
> something simple I'm overlooking.
>
> The relevant portions of my code, the schema I'm using, and the stack trace
> are at https://gist.github.com/3996847.
>
> I'm using Hadoop 0.20.2 and Avro 1.7.1 as part of CDH4.
>
> To briefly describe what I'm doing: the mapper (not included in the gist) is
> taking a bam file and spitting out some information. The key is the
> chromosome and position colon delimited and the value is an integer.
>
> The reducer is summing up all the integers at a particular position and
> creating a Pair object containing a record using the schema included in my
> gist. The second portion of the pair is an integer that I don't care
> about... if I didn't use a Pair here, I'd get an error. If this is something
> I could do differently, please correct me. :)
>
> Every time this is run, I get the stack trace included in the gist. I've run
> out of things to try to fix this... I'd really really appreciate any help I
> can get. Thanks!
>
>
>

Re: run time error during reduce stage: No field named ____ in: null

Brian Derickson 2012-11-02, 15:55

I've made another gist for this rather than clutter up the mail with code
snippets: https://gist.github.com/4002132

I basically just changed all instances of Pair<GenericRecord, Integer> in
the reducer with just GenericRecord. I also changed the output schema that
gets set in the Main function.

When I run this, I get a run time error that's also included in the above
gist: "java.lang.IllegalArgumentException: Not a Pair schema:"

The pom.xml file I'm using is also in this gist, in case I'm screwing up a
version somewhere. My intent is to be running on CDH4 using MRv1 and Avro
1.7.1, and as far as I can tell from the pom.xml I'm doing just that. Could
be mistaken.

Thanks again for your time.
On Thu, Nov 1, 2012 at 5:49 PM, Dave Beech <[EMAIL PROTECTED]> wrote:

> Hi Brian
>
> I don't think the output from the reducer should be a Pair. You said
> you got an error when you didn't use a Pair here - what was it?
>
> Cheers,
> Dave
>
> On 1 November 2012 22:09, Brian Derickson <[EMAIL PROTECTED]> wrote:
> > I've been pulling my hair out over this all day, and I'm hoping this is
> > something simple I'm overlooking.
> >
> > The relevant portions of my code, the schema I'm using, and the stack
> trace
> > are at https://gist.github.com/3996847.
> >
> > I'm using Hadoop 0.20.2 and Avro 1.7.1 as part of CDH4.
> >
> > To briefly describe what I'm doing: the mapper (not included in the
> gist) is
> > taking a bam file and spitting out some information. The key is the
> > chromosome and position colon delimited and the value is an integer.
> >
> > The reducer is summing up all the integers at a particular position and
> > creating a Pair object containing a record using the schema included in
> my
> > gist. The second portion of the pair is an integer that I don't care
> > about... if I didn't use a Pair here, I'd get an error. If this is
> something
> > I could do differently, please correct me. :)
> >
> > Every time this is run, I get the stack trace included in the gist. I've
> run
> > out of things to try to fix this... I'd really really appreciate any
> help I
> > can get. Thanks!
> >
> >
> >
>

Re: run time error during reduce stage: No field named ____ in: null

Dave Beech 2012-11-02, 16:06

I think I have it.

Your reducer isn't being called at all, because the signature of the
reducer method doesn't match the one in AvroReducer. So, the base
implementation isn't being overridden. You've stated "Iterator" where
it should actually be "Iterable".

If you use Eclipse, look for a green arrow icon next to the method
declaration - that means it's being overridden properly.

Dave

On 2 November 2012 15:55, Brian Derickson <[EMAIL PROTECTED]> wrote:
> I've made another gist for this rather than clutter up the mail with code
> snippets: https://gist.github.com/4002132
>
> I basically just changed all instances of Pair<GenericRecord, Integer> in
> the reducer with just GenericRecord. I also changed the output schema that
> gets set in the Main function.
>
> When I run this, I get a run time error that's also included in the above
> gist: "java.lang.IllegalArgumentException: Not a Pair schema:"
>
> The pom.xml file I'm using is also in this gist, in case I'm screwing up a
> version somewhere. My intent is to be running on CDH4 using MRv1 and Avro
> 1.7.1, and as far as I can tell from the pom.xml I'm doing just that. Could
> be mistaken.
>
> Thanks again for your time.
>
>
>
> On Thu, Nov 1, 2012 at 5:49 PM, Dave Beech <[EMAIL PROTECTED]> wrote:
>>
>> Hi Brian
>>
>> I don't think the output from the reducer should be a Pair. You said
>> you got an error when you didn't use a Pair here - what was it?
>>
>> Cheers,
>> Dave
>>
>> On 1 November 2012 22:09, Brian Derickson <[EMAIL PROTECTED]> wrote:
>> > I've been pulling my hair out over this all day, and I'm hoping this is
>> > something simple I'm overlooking.
>> >
>> > The relevant portions of my code, the schema I'm using, and the stack
>> > trace
>> > are at https://gist.github.com/3996847.
>> >
>> > I'm using Hadoop 0.20.2 and Avro 1.7.1 as part of CDH4.
>> >
>> > To briefly describe what I'm doing: the mapper (not included in the
>> > gist) is
>> > taking a bam file and spitting out some information. The key is the
>> > chromosome and position colon delimited and the value is an integer.
>> >
>> > The reducer is summing up all the integers at a particular position and
>> > creating a Pair object containing a record using the schema included in
>> > my
>> > gist. The second portion of the pair is an integer that I don't care
>> > about... if I didn't use a Pair here, I'd get an error. If this is
>> > something
>> > I could do differently, please correct me. :)
>> >
>> > Every time this is run, I get the stack trace included in the gist. I've
>> > run
>> > out of things to try to fix this... I'd really really appreciate any
>> > help I
>> > can get. Thanks!
>> >
>> >
>> >
>
>

Re: run time error during reduce stage: No field named ____ in: null

Brian Derickson 2012-11-02, 16:25

That did it! I never would have found that, thank you so much. This is what
I get for trying to just use Vim and Maven instead of a proper IDE. I'll
work on getting Eclipse set up.

Again, thanks a bunch. I've been pouring over this for awhile now and I'm
both glad and embarrassed it was so simple.
On Fri, Nov 2, 2012 at 11:06 AM, Dave Beech <[EMAIL PROTECTED]> wrote:

> I think I have it.
>
> Your reducer isn't being called at all, because the signature of the
> reducer method doesn't match the one in AvroReducer. So, the base
> implementation isn't being overridden. You've stated "Iterator" where
> it should actually be "Iterable".
>
> If you use Eclipse, look for a green arrow icon next to the method
> declaration - that means it's being overridden properly.
>
> Dave
>
> On 2 November 2012 15:55, Brian Derickson <[EMAIL PROTECTED]> wrote:
> > I've made another gist for this rather than clutter up the mail with code
> > snippets: https://gist.github.com/4002132
> >
> > I basically just changed all instances of Pair<GenericRecord, Integer> in
> > the reducer with just GenericRecord. I also changed the output schema
> that
> > gets set in the Main function.
> >
> > When I run this, I get a run time error that's also included in the above
> > gist: "java.lang.IllegalArgumentException: Not a Pair schema:"
> >
> > The pom.xml file I'm using is also in this gist, in case I'm screwing up
> a
> > version somewhere. My intent is to be running on CDH4 using MRv1 and Avro
> > 1.7.1, and as far as I can tell from the pom.xml I'm doing just that.
> Could
> > be mistaken.
> >
> > Thanks again for your time.
> >
> >
> >
> > On Thu, Nov 1, 2012 at 5:49 PM, Dave Beech <[EMAIL PROTECTED]> wrote:
> >>
> >> Hi Brian
> >>
> >> I don't think the output from the reducer should be a Pair. You said
> >> you got an error when you didn't use a Pair here - what was it?
> >>
> >> Cheers,
> >> Dave
> >>
> >> On 1 November 2012 22:09, Brian Derickson <[EMAIL PROTECTED]>
> wrote:
> >> > I've been pulling my hair out over this all day, and I'm hoping this
> is
> >> > something simple I'm overlooking.
> >> >
> >> > The relevant portions of my code, the schema I'm using, and the stack
> >> > trace
> >> > are at https://gist.github.com/3996847.
> >> >
> >> > I'm using Hadoop 0.20.2 and Avro 1.7.1 as part of CDH4.
> >> >
> >> > To briefly describe what I'm doing: the mapper (not included in the
> >> > gist) is
> >> > taking a bam file and spitting out some information. The key is the
> >> > chromosome and position colon delimited and the value is an integer.
> >> >
> >> > The reducer is summing up all the integers at a particular position
> and
> >> > creating a Pair object containing a record using the schema included
> in
> >> > my
> >> > gist. The second portion of the pair is an integer that I don't care
> >> > about... if I didn't use a Pair here, I'd get an error. If this is
> >> > something
> >> > I could do differently, please correct me. :)
> >> >
> >> > Every time this is run, I get the stack trace included in the gist.
> I've
> >> > run
> >> > out of things to try to fix this... I'd really really appreciate any
> >> > help I
> >> > can get. Thanks!
> >> >
> >> >
> >> >
> >
> >
>

Re: run time error during reduce stage: No field named ____ in: null

Harsh J 2012-11-02, 17:08

You could also use an @Override to assert an override at compile-time.

On Fri, Nov 2, 2012 at 9:55 PM, Brian Derickson <[EMAIL PROTECTED]> wrote:
> That did it! I never would have found that, thank you so much. This is what
> I get for trying to just use Vim and Maven instead of a proper IDE. I'll
> work on getting Eclipse set up.
>
> Again, thanks a bunch. I've been pouring over this for awhile now and I'm
> both glad and embarrassed it was so simple.
>
>
>
> On Fri, Nov 2, 2012 at 11:06 AM, Dave Beech <[EMAIL PROTECTED]> wrote:
>>
>> I think I have it.
>>
>> Your reducer isn't being called at all, because the signature of the
>> reducer method doesn't match the one in AvroReducer. So, the base
>> implementation isn't being overridden. You've stated "Iterator" where
>> it should actually be "Iterable".
>>
>> If you use Eclipse, look for a green arrow icon next to the method
>> declaration - that means it's being overridden properly.
>>
>> Dave
>>
>> On 2 November 2012 15:55, Brian Derickson <[EMAIL PROTECTED]> wrote:
>> > I've made another gist for this rather than clutter up the mail with
>> > code
>> > snippets: https://gist.github.com/4002132
>> >
>> > I basically just changed all instances of Pair<GenericRecord, Integer>
>> > in
>> > the reducer with just GenericRecord. I also changed the output schema
>> > that
>> > gets set in the Main function.
>> >
>> > When I run this, I get a run time error that's also included in the
>> > above
>> > gist: "java.lang.IllegalArgumentException: Not a Pair schema:"
>> >
>> > The pom.xml file I'm using is also in this gist, in case I'm screwing up
>> > a
>> > version somewhere. My intent is to be running on CDH4 using MRv1 and
>> > Avro
>> > 1.7.1, and as far as I can tell from the pom.xml I'm doing just that.
>> > Could
>> > be mistaken.
>> >
>> > Thanks again for your time.
>> >
>> >
>> >
>> > On Thu, Nov 1, 2012 at 5:49 PM, Dave Beech <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hi Brian
>> >>
>> >> I don't think the output from the reducer should be a Pair. You said
>> >> you got an error when you didn't use a Pair here - what was it?
>> >>
>> >> Cheers,
>> >> Dave
>> >>
>> >> On 1 November 2012 22:09, Brian Derickson <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > I've been pulling my hair out over this all day, and I'm hoping this
>> >> > is
>> >> > something simple I'm overlooking.
>> >> >
>> >> > The relevant portions of my code, the schema I'm using, and the stack
>> >> > trace
>> >> > are at https://gist.github.com/3996847.
>> >> >
>> >> > I'm using Hadoop 0.20.2 and Avro 1.7.1 as part of CDH4.
>> >> >
>> >> > To briefly describe what I'm doing: the mapper (not included in the
>> >> > gist) is
>> >> > taking a bam file and spitting out some information. The key is the
>> >> > chromosome and position colon delimited and the value is an integer.
>> >> >
>> >> > The reducer is summing up all the integers at a particular position
>> >> > and
>> >> > creating a Pair object containing a record using the schema included
>> >> > in
>> >> > my
>> >> > gist. The second portion of the pair is an integer that I don't care
>> >> > about... if I didn't use a Pair here, I'd get an error. If this is
>> >> > something
>> >> > I could do differently, please correct me. :)
>> >> >
>> >> > Every time this is run, I get the stack trace included in the gist.
>> >> > I've
>> >> > run
>> >> > out of things to try to fix this... I'd really really appreciate any
>> >> > help I
>> >> > can get. Thanks!
>> >> >
>> >> >
>> >> >
>> >
>> >
>
>

--
Harsh J