Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Re: Hive query problem on S3 table


Copy link to this message
-
Re: Hive query problem on S3 table
Tim,
  Could you try doing
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
before running the query?

Shrikanth
On Apr 18, 2013, at 8:09 AM, Tim Bittersohl wrote:

> Thanks for your answer, I tested the program with an S3N setup and unfortunately got the same error behavior...
>  
>  
> Von: Dean Wampler [mailto:[EMAIL PROTECTED]]
> Gesendet: Donnerstag, 18. April 2013 16:25
> An: [EMAIL PROTECTED]
> Betreff: Re: Hive query problem on S3 table
>  
> I'm not sure what's happening here, but one suggestion; use s3n://... instead of s3://... The "new" version is supposed to provide better performance.
>  
> dean
>  
>
> On Thu, Apr 18, 2013 at 8:43 AM, Tim Bittersohl <[EMAIL PROTECTED]> wrote:
> Hi,
>  
> I just found out, that I don't have to change the default file system of Hadoop.
> The location in the create table command has just to be changed:
>  
> CREATE EXTERNAL TABLE testtable(nyseVal STRING, cliVal STRING, dateVal STRING, number1Val STRING)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\t'
> LINES TERMINATED BY '\\n'
> STORED AS TextFile LOCATION "s3://hadoop-bucket/data/"
>  
>  
> But when I try to access the table with a command that creates a Hadoop job, I get the following error:
>  
> 13/04/18 15:29:36 ERROR security.UserGroupInformation: PriviledgedActionException as:tim (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /data/NYSE_daily.txt
> java.io.FileNotFoundException: File does not exist: /data/NYSE_daily.txt
>                 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
>                 at org.apache.hadoop.mapred.lib.CombineFileInputFormat$OneFileInfo.<init>(CombineFileInputFormat.java:462)
>                 at org.apache.hadoop.mapred.lib.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:256)
>                 at org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:212)
>                 at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:411)
>                 at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:377)
>                 at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:387)
>                 at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1091)
>                 at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1083)
>                 at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
>                 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:993)
>                 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:946)
>                 at java.security.AccessController.doPrivileged(Native Method)
>                 at javax.security.auth.Subject.doAs(Subject.java:415)
>                 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>                 at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:946)
>                 at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:920)
>                 at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447)
>                 at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
>                 at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
>                 at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>                 at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1352)
>                 at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)
>                 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
>                 at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
>                 at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644)
+
Tim Bittersohl 2013-04-18, 13:43
+
Dean Wampler 2013-04-18, 14:25
+
Panshul Whisper 2013-04-18, 14:18
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB