Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - run time error during reduce stage: No field named ____ in: null


Copy link to this message
-
Re: run time error during reduce stage: No field named ____ in: null
Harsh J 2012-11-02, 17:08
You could also use an @Override to assert an override at compile-time.

On Fri, Nov 2, 2012 at 9:55 PM, Brian Derickson <[EMAIL PROTECTED]> wrote:
> That did it! I never would have found that, thank you so much. This is what
> I get for trying to just use Vim and Maven instead of a proper IDE. I'll
> work on getting Eclipse set up.
>
> Again, thanks a bunch. I've been pouring over this for awhile now and I'm
> both glad and embarrassed it was so simple.
>
>
>
> On Fri, Nov 2, 2012 at 11:06 AM, Dave Beech <[EMAIL PROTECTED]> wrote:
>>
>> I think I have it.
>>
>> Your reducer isn't being called at all, because the signature of the
>> reducer method doesn't match the one in AvroReducer. So, the base
>> implementation isn't being overridden. You've stated "Iterator" where
>> it should actually be "Iterable".
>>
>> If you use Eclipse, look for a green arrow icon next to the method
>> declaration - that means it's being overridden properly.
>>
>> Dave
>>
>> On 2 November 2012 15:55, Brian Derickson <[EMAIL PROTECTED]> wrote:
>> > I've made another gist for this rather than clutter up the mail with
>> > code
>> > snippets: https://gist.github.com/4002132
>> >
>> > I basically just changed all instances of Pair<GenericRecord, Integer>
>> > in
>> > the reducer with just GenericRecord. I also changed the output schema
>> > that
>> > gets set in the Main function.
>> >
>> > When I run this, I get a run time error that's also included in the
>> > above
>> > gist: "java.lang.IllegalArgumentException: Not a Pair schema:"
>> >
>> > The pom.xml file I'm using is also in this gist, in case I'm screwing up
>> > a
>> > version somewhere. My intent is to be running on CDH4 using MRv1 and
>> > Avro
>> > 1.7.1, and as far as I can tell from the pom.xml I'm doing just that.
>> > Could
>> > be mistaken.
>> >
>> > Thanks again for your time.
>> >
>> >
>> >
>> > On Thu, Nov 1, 2012 at 5:49 PM, Dave Beech <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hi Brian
>> >>
>> >> I don't think the output from the reducer should be a Pair. You said
>> >> you got an error when you didn't use a Pair here - what was it?
>> >>
>> >> Cheers,
>> >> Dave
>> >>
>> >> On 1 November 2012 22:09, Brian Derickson <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > I've been pulling my hair out over this all day, and I'm hoping this
>> >> > is
>> >> > something simple I'm overlooking.
>> >> >
>> >> > The relevant portions of my code, the schema I'm using, and the stack
>> >> > trace
>> >> > are at https://gist.github.com/3996847.
>> >> >
>> >> > I'm using Hadoop 0.20.2 and Avro 1.7.1 as part of CDH4.
>> >> >
>> >> > To briefly describe what I'm doing: the mapper (not included in the
>> >> > gist) is
>> >> > taking a bam file and spitting out some information. The key is the
>> >> > chromosome and position colon delimited and the value is an integer.
>> >> >
>> >> > The reducer is summing up all the integers at a particular position
>> >> > and
>> >> > creating a Pair object containing a record using the schema included
>> >> > in
>> >> > my
>> >> > gist. The second portion of the pair is an integer that I don't care
>> >> > about... if I didn't use a Pair here, I'd get an error. If this is
>> >> > something
>> >> > I could do differently, please correct me. :)
>> >> >
>> >> > Every time this is run, I get the stack trace included in the gist.
>> >> > I've
>> >> > run
>> >> > out of things to try to fix this... I'd really really appreciate any
>> >> > help I
>> >> > can get. Thanks!
>> >> >
>> >> >
>> >> >
>> >
>> >
>
>

--
Harsh J