Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Map output records/reducer input records mismatch


Copy link to this message
-
Re: Map output records/reducer input records mismatch
Hi, Scott,

thanks for your reply.

> What Avro version is this happening with? What JVM version?

We are using Avro 1.5.1 and Sun JDK 6, but the exact version I will have to look up.

>
> On a hunch, have you tried adding -XX:-UseLoopPredicate to the JVM args if
> it is Sun and JRE 6u21 or later? (some issues in loop predicates affect
> Java 6 too, just not as many as the recent news on Java7).
>
> Otherwise, it may likely be the same thing as AVRO-782.  Any extra
> information related to that issue would be welcome.

I will have to collect it. In the meanwhile, do you have any reasonable explanations of the issue besides it being something like AVRO-782?

Thanks a lot,
Vyacheslav

>
> Thanks!
>
> -Scott
>
>
>
> On 8/16/11 8:39 AM, "Vyacheslav Zholudev" <[EMAIL PROTECTED]>
> wrote:
>
>> Hi,
>>
>> I'm having multiple hadoop jobs that use the avro mapred API.
>> Only in one of the jobs I have a visible mismatch between a number of map
>> output records and reducer input records.
>>
>> Does anybody encountered such a behavior? Can anybody think of possible
>> explanations of this phenomenon?
>>
>> Any pointers/thoughts are highly appreciated!
>>
>> Best,
>> Vyacheslav
>
>

Best,
Vyacheslav
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB