Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Hive query problem on S3 table


Copy link to this message
-
Re: Hive query problem on S3 table
shrikanth shankar 2013-04-18, 15:32
Tim,
  Could you try doing
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
before running the query?

Shrikanth
On Apr 18, 2013, at 8:09 AM, Tim Bittersohl wrote:

> Thanks for your answer, I tested the program with an S3N setup and unfortunately got the same error behavior...
>  
>  
> Von: Dean Wampler [mailto:[EMAIL PROTECTED]]
> Gesendet: Donnerstag, 18. April 2013 16:25
> An: [EMAIL PROTECTED]
> Betreff: Re: Hive query problem on S3 table
>  
> I'm not sure what's happening here, but one suggestion; use s3n://... instead of s3://... The "new" version is supposed to provide better performance.
>  
> dean
>  
>
> On Thu, Apr 18, 2013 at 8:43 AM, Tim Bittersohl <[EMAIL PROTECTED]> wrote:
> Hi,
>  
> I just found out, that I don't have to change the default file system of Hadoop.
> The location in the create table command has just to be changed:
>  
> CREATE EXTERNAL TABLE testtable(nyseVal STRING, cliVal STRING, dateVal STRING, number1Val STRING)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\t'
> LINES TERMINATED BY '\\n'
> STORED AS TextFile LOCATION "s3://hadoop-bucket/data/"
>  
>  
> But when I try to access the table with a command that creates a Hadoop job, I get the following error:
>  
> 13/04/18 15:29:36 ERROR security.UserGroupInformation: PriviledgedActionException as:tim (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /data/NYSE_daily.txt
> java.io.FileNotFoundException: File does not exist: /data/NYSE_daily.txt
>                 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:807)
>                 at org.apache.hadoop.mapred.lib.CombineFileInputFormat$OneFileInfo.<init>(CombineFileInputFormat.java:462)
>                 at org.apache.hadoop.mapred.lib.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:256)
>                 at org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:212)
>                 at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:411)
>                 at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:377)
>                 at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:387)
>                 at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1091)
>                 at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1083)
>                 at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
>                 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:993)
>                 at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:946)
>                 at java.security.AccessController.doPrivileged(Native Method)
>                 at javax.security.auth.Subject.doAs(Subject.java:415)
>                 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>                 at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:946)
>                 at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:920)
>                 at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447)
>                 at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
>                 at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
>                 at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>                 at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1352)
>                 at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)
>                 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
>                 at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
>                 at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644)