Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Re: FileSystem Closed.


+
Harsh J 2012-07-10, 16:39
Copy link to this message
-
Re: FileSystem Closed.
Can you share your query and use case?

~Aniket

On Tue, Jul 10, 2012 at 9:39 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> This appears to be a Hive issue (something probably called FS.close()
> too early?). Redirecting to the Hive user lists as they can help
> better with this.
>
> On Tue, Jul 10, 2012 at 9:59 PM, 안의건 <[EMAIL PROTECTED]> wrote:
> > Hello. I have a problem with the filesystem closing.
> >
> > The filesystem was closed when the hive query is running.
> > It is 'select' query and the data size is about 1TB.
> > I'm using hadoop-0.20.2 and hive-0.7.1.
> >
> > The error log is telling that tmp file is not deleted, or the tmp path
> > exception is occurred.
> >
> > Is there any hadoop configuration I'm missing?
> >
> > Thank you
> >
> > [stderr logs]
> > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> > Filesystem closed
> > at
> >
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:454)
> > at
> >
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:636)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:557)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> > at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.io.IOException: Filesystem closed
> > at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:226)
> > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:617)
> > at
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:453)
> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:648)
> > at org.apache.hadoop.fs.FileSystem.deleteOnExit(FileSystem.java:615)
> > at
> >
> org.apache.hadoop.hive.shims.Hadoop20Shims.fileSystemDeleteOnExit(Hadoop20Shims.java:68)
> > at
> >
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:451)
> > ... 12 more
>
>
>
> --
> Harsh J
>

--
"...:::Aniket:::... Quetzalco@tl"
+
안의건 2012-07-11, 00:09