Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> BytesWritable


Copy link to this message
-
Re: BytesWritable
Hi Josh,
it doesn't have to. However, your table does not specify any input format,
which makes Hive use the default HiveIgnoreKeyTextOutputFormat (see
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop.hive/hive-exec/0.7.1-cdh3u1/org/apache/hadoop/hive/ql/io/HiveIgnoreKeyTextOutputFormat.java
).

What's the bigger picture, meaning, what is the actual format you want to
write down to disk ?

If you're storing it in text format, your SerDe's serialize method has to
return either a Text or a BytesWritable, as you can see in the code linked
earlier. It looks like it's returning a DocumentWritable object, which
HiveIgnoreKeyTextOutputFormat does not know how to handle.

R.
On Mon, Jul 29, 2013 at 5:42 AM, Clum, Josh <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I wrote a custom serde that stores takes data and stores it in objects
> called DocumentWritable that inherits from Writable. When I try to run code
> to insert into the table that uses my serde, I get an error saying that
> DocumentWritable cannot be cast to BytesWritable. Does my DocumentWritable
> need to inherit from org.apache.hadoop.io.BytesWritable?
>
> *I'm creating a table using a custom serde that I wrote:*
>
> CREATE TABLE avro_test (
>     NOTEBOOK_ID STRING,
>    …
>     DLC_XID STRING
>   )
> ROW FORMAT SERDE 'com.merck.huron.hive.LuceneSerde';
>
> *When I try to insert data using a select statement I am storing it into
> a class I have called DocumentWritable:*
>
> INSERT OVERWRITE TABLE avro_test SELECT * FROM avro
>
> *DocumentWritable inherits from Writable. When I do the select and my
> serde code gets executed I get the following error:*
>
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> {"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO
> DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL
> SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null}
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> {"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO
> DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL
> SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null}
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:565)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
> ... 8 more
> Caused by: java.lang.ClassCastException: com.comp.proj.indexing.*DocumentWritable
> cannot be cast to org.apache.hadoop.io.BytesWritable*
> at
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat$1.write(HiveIgnoreKeyTextOutputFormat.java:90)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:606)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)

Good judgement comes with experience.
Experience comes with bad judgement.
Roberto Congiu - Data Engineer - OpenX
tel: +1 626 466 1141
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB