Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Hive 0.11.0 | Issue with ORC Tables


Copy link to this message
-
Re: Hive 0.11.0 | Issue with ORC Tables
Nitin Pawar 2013-09-20, 11:19
Keshav,

Owen has provided the solution already. Thats the easiest of the the lot
and from the master who wrote ORC himself :)

to put it in simple words what he has suggested is,

create a staging table which will be based on default text data format.
>From the staging data load data into a ORC file backed table.

you can refer Owen's mail for the respective queries.

On Fri, Sep 20, 2013 at 4:46 PM, Savant, Keshav <
[EMAIL PROTECTED]> wrote:

>  Hi Nitin,****
>
> ** **
>
> Thanks for your reply,  we were in an impression that the codec will be
> responsible for ORC format conversion also.****
>
> However as per your reply it seems that a conversion from normal CSV to
> ORC is required before hive upload.****
>
> ** **
>
> We got some leads from following URLs****
>
> https://cwiki.apache.org/Hive/languagemanual-orc.html****
>
> http://www.math.uic.edu/t3m/SnapPy/installing.html****
>
> ** **
>
> Please suggest how it can be done using some already available libraries,
> or we need to write our own converter.****
>
> ** **
>
> Kind Regards,****
>
> Keshav****
>
> ** **
>
> *From:* Nitin Pawar [mailto:[EMAIL PROTECTED]]
> *Sent:* Thursday, September 19, 2013 5:56 PM
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: Hive 0.11.0 | Issue with ORC Tables****
>
> ** **
>
> How did you create "test.txt" as ORC file? ****
>
> ** **
>
> ** **
>
> On Thu, Sep 19, 2013 at 5:34 PM, Savant, Keshav <
> [EMAIL PROTECTED]> wrote:****
>
> Hi All,****
>
>  ****
>
> We have setup apache “hive 0.11.0” services on Hadoop cluster (apache
> version 0.20.203.0). Hive is showing expected results when tables are
> stored as *TextFile*. ****
>
> Whereas, Hive 0.11.0’s new feature ORC(*Optimized Row Columnar*) is
> throwing an exception while running a select query, when we run select
> queries on tables stored as “*ORC*”.****
>
> Stacktrace of the exception :****
>
>  ****
>
> 2013-09-19 20:33:38,095 ERROR CliDriver
> (SessionState.java:printError(386)) - Failed with exception
> java.io.IOException:com.google.protobuf.InvalidProtocolBufferException:
> While parsing a protocol message, the input ended unexpectedly in the
> middle of a field.  This could mean either than the input has been
> truncated or that an embedded message misreported its own length.****
>
> java.io.IOException: com.google.protobuf.InvalidProtocolBufferException:
> While parsing a protocol message, the input ended unexpectedly in the
> middle of a field.  This could mean either than the input has been
> truncated or that an embedded message misreported its own length.****
>
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
> ****
>
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
> ****
>
>         at
> org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)****
>
>         at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)**
> **
>
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)**
> **
>
>         at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)****
>
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)****
>
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)***
> *
>
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)**
> **
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)****
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> ****
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> ****
>
>         at java.lang.reflect.Method.invoke(Method.java:597)****
>
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)****
>
> Caused by: com.google.protobuf.InvalidProtocolBufferException: While
> parsing a protocol message, the input ended unexpectedly in the middle of a
> field.  This could mean either than the input has been truncated or that an

Nitin Pawar