Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Hive 0.11.0 | Issue with ORC Tables


+
Savant, Keshav 2013-09-19, 12:04
+
Owen OMalley 2013-09-19, 16:22
+
Nitin Pawar 2013-09-19, 12:25
+
Savant, Keshav 2013-09-20, 11:16
+
Nitin Pawar 2013-09-20, 11:19
Copy link to this message
-
Re: Hive 0.11.0 | Issue with ORC Tables
Another advantage to the method described by Owen is your process of
creating the ORC file is distributed.  (Rather than precreating the ORC
file off cluster and then moving into the cluster).  This way, you just
push your text files into the cluster, do the select statement and push
into the ORC table. This is also a great opportunity to add partitioning,
sorting, or enrich the data at load time.
On Fri, Sep 20, 2013 at 6:19 AM, Nitin Pawar <[EMAIL PROTECTED]>wrote:

> Keshav,
>
> Owen has provided the solution already. Thats the easiest of the the lot
> and from the master who wrote ORC himself :)
>
> to put it in simple words what he has suggested is,
>
> create a staging table which will be based on default text data format.
> From the staging data load data into a ORC file backed table.
>
> you can refer Owen's mail for the respective queries.
>
>
>
> On Fri, Sep 20, 2013 at 4:46 PM, Savant, Keshav <
> [EMAIL PROTECTED]> wrote:
>
>>  Hi Nitin,****
>>
>> ** **
>>
>> Thanks for your reply,  we were in an impression that the codec will be
>> responsible for ORC format conversion also.****
>>
>> However as per your reply it seems that a conversion from normal CSV to
>> ORC is required before hive upload.****
>>
>> ** **
>>
>> We got some leads from following URLs****
>>
>> https://cwiki.apache.org/Hive/languagemanual-orc.html****
>>
>> http://www.math.uic.edu/t3m/SnapPy/installing.html****
>>
>> ** **
>>
>> Please suggest how it can be done using some already available libraries,
>> or we need to write our own converter.****
>>
>> ** **
>>
>> Kind Regards,****
>>
>> Keshav****
>>
>> ** **
>>
>> *From:* Nitin Pawar [mailto:[EMAIL PROTECTED]]
>> *Sent:* Thursday, September 19, 2013 5:56 PM
>> *To:* [EMAIL PROTECTED]
>> *Subject:* Re: Hive 0.11.0 | Issue with ORC Tables****
>>
>> ** **
>>
>> How did you create "test.txt" as ORC file? ****
>>
>> ** **
>>
>> ** **
>>
>> On Thu, Sep 19, 2013 at 5:34 PM, Savant, Keshav <
>> [EMAIL PROTECTED]> wrote:****
>>
>> Hi All,****
>>
>>  ****
>>
>> We have setup apache “hive 0.11.0” services on Hadoop cluster (apache
>> version 0.20.203.0). Hive is showing expected results when tables are
>> stored as *TextFile*. ****
>>
>> Whereas, Hive 0.11.0’s new feature ORC(*Optimized Row Columnar*) is
>> throwing an exception while running a select query, when we run select
>> queries on tables stored as “*ORC*”.****
>>
>> Stacktrace of the exception :****
>>
>>  ****
>>
>> 2013-09-19 20:33:38,095 ERROR CliDriver
>> (SessionState.java:printError(386)) - Failed with exception
>> java.io.IOException:com.google.protobuf.InvalidProtocolBufferException:
>> While parsing a protocol message, the input ended unexpectedly in the
>> middle of a field.  This could mean either than the input has been
>> truncated or that an embedded message misreported its own length.****
>>
>> java.io.IOException: com.google.protobuf.InvalidProtocolBufferException:
>> While parsing a protocol message, the input ended unexpectedly in the
>> middle of a field.  This could mean either than the input has been
>> truncated or that an embedded message misreported its own length.****
>>
>>         at
>> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544)
>> ****
>>
>>         at
>> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488)
>> ****
>>
>>         at
>> org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)****
>>
>>         at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1412)*
>> ***
>>
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)*
>> ***
>>
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)****
>>
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)****
>>
>>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)**
>> **
>>
>>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB