Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Assertions in reducer when running hive query


Copy link to this message
-
RE: Assertions in reducer when running hive query
Can you help reduce the test case a bit. Does it happpen if you just run the subquery as well?

Ashish

________________________________
From: Eva Tse [mailto:[EMAIL PROTECTED]]
Sent: Thursday, August 27, 2009 7:08 PM
To: [EMAIL PROTECTED]
Subject: Assertions in reducer when running hive query

We are running a Hive query like the following:

from (select sum (c.countrow) as countrow, c.c_id as c_id, f.dt_id as dt_id, f.xid as xid, f.esn as esn, f.hour as hour from session_facts f join session_cds c on (f.xid = c.xid and f.esn = c.esn) where c.dateint = 20090825 and f.dateint = 20090825 and c.hour = f.hour and c.c_id is not null group by c.c_id, f.dt_id, f.xid, f.esn, f.hour) x
insert overwrite table temp_a select count (1), x.c_id, x.dt_id, x.hour where x.countrow > 0 group by x.c_id, x.dt_id, x.hour
insert overwrite table temp_b select count (1), x.c_id, x.dt_id, x.hour group by x.c_id, x.dt_id, x.hour

We are getting the following exceptions in our reducers after we turn on the flag -enableassertions in mapred.child.java.opts.  If we don't turn on the flag, it is fine.

Have anyone seen this before and do we know why? We are using Hive truck version # 795559 with Hadoop 0.20.
Thanks,
Eva.

2009-08-27 18:20:39,782 INFO org.apache.hadoop.mapred.Merger: Merging 5 sorted segments
2009-08-27 18:20:39,794 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 5 segments left of total size: 264428308 bytes
2009-08-27 18:20:39,830 INFO ExecReducer: conf classpath = [file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/classes, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/attempt_200908212037_4391_r_000000_1/]
2009-08-27 18:20:39,830 INFO ExecReducer: thread classpath = [file:/etc/hadoop-0.20/conf.empty/, file:/usr/java/jdk1.6.0_07/lib/tools.jar, file:/usr/lib/hadoop-0.20/, file:/usr/lib/hadoop-0.20/hadoop-0.20.0~17-1.cloudera-core.jar, file:/usr/lib/hadoop-0.20/lib/commons-cli-2.0-SNAPSHOT.jar, file:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar, file:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar, file:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar, file:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar, file:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar, file:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar, file:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar, file:/usr/lib/hadoop-0.20/lib/hadoop-0.20.0~17-1.cloudera-fairscheduler.jar, file:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar, file:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar, file:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar, file:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar, file:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar, file:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar, file:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar, file:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar, file:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar, file:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar, file:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar, file:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar, file:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar, file:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar, file:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar, file:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/attempt_200908212037_4391_r_000000_1/work/, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/classes, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/jars/, file:/mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908212037_4391/attempt_200908212037_4391_r_000000_1/work/]
2009-08-27 18:20:40,697 INFO ExecReducer:
<JOIN>Id =6
  <Children>
    <FIL>Id =7
      <Children>
        <SEL>Id =8
          <Children>
            <GBY>Id =9
              <Children>
                <FS>Id =10
                  <Parent>Id = 9 <\Parent>
                <\FS>
              <\Children>
              <Parent>Id = 8 <\Parent>
            <\GBY>
          <\Children>
          <Parent>Id = 7 <\Parent>
        <\SEL>
      <\Children>
      <Parent>Id = 6 <\Parent>
    <\FIL>
  <\Children>
<\JOIN>
2009-08-27 18:20:40,698 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: Initializing Self 6 JOIN
2009-08-27 18:20:40,700 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator: COMMONJOIN struct<key:struct<joinkey0:string,joinkey1:string>,value:struct<_col0:string,_col1:string,_col2:int,_col24:string,_col25:string>,alias:tinyint>
2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator: JOIN struct<_col0:string,_col1:string,_col2:int,_col24:string,_col25:string,_col28:int,_col33:int,_col34:string,_col35:string> totalsz = 9
2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: Operator 6 JOIN initialized
2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: Initializing children of 6 JOIN
2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing child 7 FIL
2009-08-27 18:20:40,710 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing Self 7 FIL
2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Operator 7 FIL initialized
2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing children of 7 FIL
2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing child 8 SEL
2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing Self 8 SEL
2009-08-27 18:20:40,716 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT struct<_col0:string,_col1:string,_col2:int,_col24:string,_col25:string,_col28:int,_col33:int,_col34:string,_col35:string>
2009-08-27 18:20:40,717 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Operator 8 SEL initialized
2009-08-27 18:20:40,717 INFO org.apache.had
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB