Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Map output records/reducer input records mismatch


+
Vyacheslav Zholudev 2011-08-16, 15:39
+
Scott Carey 2011-08-16, 20:22
Copy link to this message
-
Re: Map output records/reducer input records mismatch
Hi, Scott,

thanks for your reply.

> What Avro version is this happening with? What JVM version?

We are using Avro 1.5.1 and Sun JDK 6, but the exact version I will have to look up.

>
> On a hunch, have you tried adding -XX:-UseLoopPredicate to the JVM args if
> it is Sun and JRE 6u21 or later? (some issues in loop predicates affect
> Java 6 too, just not as many as the recent news on Java7).
>
> Otherwise, it may likely be the same thing as AVRO-782.  Any extra
> information related to that issue would be welcome.

I will have to collect it. In the meanwhile, do you have any reasonable explanations of the issue besides it being something like AVRO-782?

Thanks a lot,
Vyacheslav

>
> Thanks!
>
> -Scott
>
>
>
> On 8/16/11 8:39 AM, "Vyacheslav Zholudev" <[EMAIL PROTECTED]>
> wrote:
>
>> Hi,
>>
>> I'm having multiple hadoop jobs that use the avro mapred API.
>> Only in one of the jobs I have a visible mismatch between a number of map
>> output records and reducer input records.
>>
>> Does anybody encountered such a behavior? Can anybody think of possible
>> explanations of this phenomenon?
>>
>> Any pointers/thoughts are highly appreciated!
>>
>> Best,
>> Vyacheslav
>
>

Best,
Vyacheslav
+
Scott Carey 2011-08-17, 01:56
+
Vyacheslav Zholudev 2011-08-17, 08:32
+
Scott Carey 2011-08-17, 17:06
+
Vyacheslav Zholudev 2011-08-17, 12:02
+
Scott Carey 2011-08-17, 17:18
+
Vyacheslav Zholudev 2011-08-17, 18:09
+
Vyacheslav Zholudev 2011-08-17, 22:02
+
Vyacheslav Zholudev 2011-08-17, 22:59
+
Scott Carey 2011-08-17, 23:47
+
Vyacheslav Zholudev 2011-08-18, 12:50
+
Vyacheslav Zholudev 2011-08-17, 15:49