Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> BytesWritable


Copy link to this message
-
Re: BytesWritable
Hi Josh,
it doesn't have to. However, your table does not specify any input format,
which makes Hive use the default HiveIgnoreKeyTextOutputFormat (see
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop.hive/hive-exec/0.7.1-cdh3u1/org/apache/hadoop/hive/ql/io/HiveIgnoreKeyTextOutputFormat.java
).

What's the bigger picture, meaning, what is the actual format you want to
write down to disk ?

If you're storing it in text format, your SerDe's serialize method has to
return either a Text or a BytesWritable, as you can see in the code linked
earlier. It looks like it's returning a DocumentWritable object, which
HiveIgnoreKeyTextOutputFormat does not know how to handle.

R.
On Mon, Jul 29, 2013 at 5:42 AM, Clum, Josh <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I wrote a custom serde that stores takes data and stores it in objects
> called DocumentWritable that inherits from Writable. When I try to run code
> to insert into the table that uses my serde, I get an error saying that
> DocumentWritable cannot be cast to BytesWritable. Does my DocumentWritable
> need to inherit from org.apache.hadoop.io.BytesWritable?
>
> *I'm creating a table using a custom serde that I wrote:*
>
> CREATE TABLE avro_test (
>     NOTEBOOK_ID STRING,
>    …
>     DLC_XID STRING
>   )
> ROW FORMAT SERDE 'com.merck.huron.hive.LuceneSerde';
>
> *When I try to insert data using a select statement I am storing it into
> a class I have called DocumentWritable:*
>
> INSERT OVERWRITE TABLE avro_test SELECT * FROM avro
>
> *DocumentWritable inherits from Writable. When I do the select and my
> serde code gets executed I get the following error:*
>
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> {"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO
> DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL
> SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null}
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> {"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO
> DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL
> SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null}
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:565)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
> ... 8 more
> Caused by: java.lang.ClassCastException: com.comp.proj.indexing.*DocumentWritable
> cannot be cast to org.apache.hadoop.io.BytesWritable*
> at
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat$1.write(HiveIgnoreKeyTextOutputFormat.java:90)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:606)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)

Good judgement comes with experience.
Experience comes with bad judgement.
Roberto Congiu - Data Engineer - OpenX
tel: +1 626 466 1141