Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Assertions in reducer when running hive query


Copy link to this message
-
RE: Assertions in reducer when running hive query
Can you help reduce the test case a bit. Does it happpen if you just run the subquery as well?

Ashish

________________________________
From: Eva Tse [mailto:[EMAIL PROTECTED]]
Sent: Thursday, August 27, 2009 7:08 PM
To: [EMAIL PROTECTED]
Subject: Assertions in reducer when running hive query

We are running a Hive query like the following:

from (select sum (c.countrow) as countrow, c.c_id as c_id, f.dt_id as dt_id, f.xid as xid, f.esn as esn, f.hour as hour from session_facts f join session_cds c on (f.xid = c.xid and f.esn = c.esn) where c.dateint = 20090825 and f.dateint = 20090825 and c.hour = f.hour and c.c_id is not null group by c.c_id, f.dt_id, f.xid, f.esn, f.hour) x
insert overwrite table temp_a select count (1), x.c_id, x.dt_id, x.hour where x.countrow > 0 group by x.c_id, x.dt_id, x.hour
insert overwrite table temp_b select count (1), x.c_id, x.dt_id, x.hour group by x.c_id, x.dt_id, x.hour

We are getting the following exceptions in our reducers after we turn on the flag -enableassertions in mapred.child.java.opts.  If we don't turn on the flag, it is fine.

Have anyone seen this before and do we know why? We are using Hive truck version # 795559 with Hadoop 0.20.
Thanks,
Eva.

2009-08-27 18:20:39,782 INFO org.apache.hadoop.mapred.Merger: Merging 5 sorted segments
2009-08-27 18:20:39,794 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 5 segments left of total size: 264428308 bytes
2009-08-27 18:20:39,830 INFO ExecReducer: conf classpath = [file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/classes, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/attempt_200908212037_4391_r_000000_1/]
2009-08-27 18:20:39,830 INFO ExecReducer: thread classpath = [file:/etc/hadoop-0.20/conf.empty/, file:/usr/java/jdk1.6.0_07/lib/tools.jar, file:/usr/lib/hadoop-0.20/, file:/usr/lib/hadoop-0.20/hadoop-0.20.0~17-1.cloudera-core.jar, file:/usr/lib/hadoop-0.20/lib/commons-cli-2.0-SNAPSHOT.jar, file:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar, file:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar, file:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar, file:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar, file:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar, file:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar, file:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar, file:/usr/lib/hadoop-0.20/lib/hadoop-0.20.0~17-1.cloudera-fairscheduler.jar, file:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar, file:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar, file:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar, file:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar, file:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar, file:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar, file:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar, file:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar, file:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar, file:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar, file:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar, file:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar, file:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar, file:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar, file:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar, file:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/attempt_200908212037_4391_r_000000_1/work/, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/classes, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/attempt_200908212037_4391_r_000000_1/work/]
2009-08-27 18:20:40,697 INFO ExecReducer:
<JOIN>Id =6
  <Children>
    <FIL>Id =7
      <Children>
        <SEL>Id =8
          <Children>
            <GBY>Id =9
              <Children>
                <FS>Id =10
                  <Parent>Id = 9 <\Parent>
                <\FS>
              <\Children>
              <Parent>Id = 8 <\Parent>
            <\GBY>
          <\Children>
          <Parent>Id = 7 <\Parent>
        <\SEL>
      <\Children>
      <Parent>Id = 6 <\Parent>
    <\FIL>
  <\Children>
<\JOIN>
2009-08-27 18:20:40,698 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: Initializing Self 6 JOIN
2009-08-27 18:20:40,700 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator: COMMONJOIN struct<key:struct<joinkey0:string,joinkey1:string>,value:struct<_col0:string,_col1:string,_col2:int,_col24:string,_col25:string>,alias:tinyint>
2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator: JOIN struct<_col0:string,_col1:string,_col2:int,_col24:string,_col25:string,_col28:int,_col33:int,_col34:string,_col35:string> totalsz = 9
2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: Operator 6 JOIN initialized
2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: Initializing children of 6 JOIN
2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing child 7 FIL
2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing Self 7 FIL
2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Operator 7 FIL initialized
2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing children of 7 FIL
2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing child 8 SEL
2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing Self 8 SEL
2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT struct<_col0:string,_col1:string,_col2:int,_col24:string,_col25:string,_col28:int,_col33:int,_col34:string,_col35:string>
2009-08-27 18:20:40,717 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Operator 8 SEL initialized
2009-08-27 18:20:40,717 INFO org.apache.had