Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - ClassCastException during reduce-side join, but not map-side join


Copy link to this message
-
ClassCastException during reduce-side join, but not map-side join
Anthony Urso 2013-01-17, 20:48
I am getting an exception when joining two tables with Amazon's Hive
0.8.1 on Amazon EMR, and I've run out of ideas on how to fix it.

The query is something along the lines of

Q1: SELECT count(*) FROM t1 x JOIN t2 y ON (x.id = y.x_id);

Which ends up throwing an exception like this in some of the mappers:

java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while
 processing row
{"T1C1":"t2c1\u0001t2c2\u0001t2c3\u0001t2c4\u0001t2c5\u0001t2c6\u0001t2c7\u0001t2c8\u0001t2c8\u0001t2c9\u0001t2c10\u0001t2c11\u0001t2c12\u0001null\u0001null\u0001null\u0001null\u0001t2c18","T1C2":null,"T1C3":null,"T1C4":null,"T1C5":null}
...
...
...
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text
cannot be cast to org.apache.hadoop
.io.IntWritable
        at org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyIntObjectInspector.get(Laz
yIntObjectInspector.java:38)
...
...

... where TxCy is the name of the yth column in the xth table, and
txcy is the value of the yth column in the xth table for this row.

It looks like the deserializer for table 2 is getting an incorrectly
formatted row from table 1 and is not splitting it on \u0001 as
appears to be intended.

Self joins and selecting from either table separately works fine and
all rows are deserialized correctly in those cases.

Here are the schemas for the tables:

CREATE EXTERNAL TABLE  t1 (
c1 STRING,
...
c5 STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LOCATION 's3n://t1/';

CREATE EXTERNAL TABLE  t2 (
c1 STRING,
...
c18 STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LOCATION 's3n://t2/';

Finally, changing Q1 above to use a map-side join:

Q2: SELECT /*+ MAPJOIN(x) */ count(*) FROM t1 x JOIN t2 y ON (x.id = y.x_id);

prevents the exception from occurring at all.

Is this a known bug in Apache Hive 0.8.1 or Amazon's 0.8.1 version?
If so, is there a fix or non-mapside workaround?

Thanks,
Anthony