Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Re: FileSystem Closed.


+
Harsh J 2012-07-10, 16:39
Copy link to this message
-
Re: FileSystem Closed.
Can you share your query and use case?

~Aniket

On Tue, Jul 10, 2012 at 9:39 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> This appears to be a Hive issue (something probably called FS.close()
> too early?). Redirecting to the Hive user lists as they can help
> better with this.
>
> On Tue, Jul 10, 2012 at 9:59 PM, 안의건 <[EMAIL PROTECTED]> wrote:
> > Hello. I have a problem with the filesystem closing.
> >
> > The filesystem was closed when the hive query is running.
> > It is 'select' query and the data size is about 1TB.
> > I'm using hadoop-0.20.2 and hive-0.7.1.
> >
> > The error log is telling that tmp file is not deleted, or the tmp path
> > exception is occurred.
> >
> > Is there any hadoop configuration I'm missing?
> >
> > Thank you
> >
> > [stderr logs]
> > org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> > Filesystem closed
> > at
> >
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:454)
> > at
> >
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:636)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:557)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> > at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.io.IOException: Filesystem closed
> > at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:226)
> > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:617)
> > at
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:453)
> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:648)
> > at org.apache.hadoop.fs.FileSystem.deleteOnExit(FileSystem.java:615)
> > at
> >
> org.apache.hadoop.hive.shims.Hadoop20Shims.fileSystemDeleteOnExit(Hadoop20Shims.java:68)
> > at
> >
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:451)
> > ... 12 more
>
>
>
> --
> Harsh J
>

--
"...:::Aniket:::... Quetzalco@tl"
+
안의건 2012-07-11, 00:09
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB