Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> How Can I store the Hive query result in one file ?


+
Matouk IFTISSEN 2013-07-04, 09:42
+
Nitin Pawar 2013-07-04, 10:38
+
Bertrand Dechoux 2013-07-04, 11:09
Copy link to this message
-
Re: How Can I store the Hive query result in one file ?
I have found that for output larger than a few GB, redirecting stdout results in an incomplete file.  For very large output, I do CREATE TABLE MYTABLE AS SELECT ... and then copy the resulting HDFS files directly out of /user/hive/warehouse.
 

________________________________
 From: Bertrand Dechoux <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Thursday, July 4, 2013 7:09 AM
Subject: Re: How Can I store the Hive query result in one file ?
  
The question is what is the volume of your output. There is one file per output task (map or reduce) because that way each can write it independently and in parallel. That's how mapreduce work. And except by forcing the number of tasks to 1, there is no certain way to have one output file.

But indeed if the volume is low enough, you could also capture the standard output into a local file like Nitin described.

Bertrand

On Thu, Jul 4, 2013 at 12:38 PM, Nitin Pawar <[EMAIL PROTECTED]> wrote:

will hive -e "query" > filename  or hive -f query.q > filename will do ? 
>
>
>you specially want it to write into a named file on hdfs only? 
>
>
>
>On Thu, Jul 4, 2013 at 3:12 PM, Matouk IFTISSEN <[EMAIL PROTECTED]> wrote:
>
>Hello Hive users,
>>Is there a manner to store the Hive  query result (SELECT *.....) in a specfique  and alone file (given the file name) like (INSERT OVERWRITE LOCAL DIRECTORY '/directory_path_name/')?
>>Thanks for your answers
>>
>>
>>
>
>
>
>--
>Nitin Pawar
>
--
Bertrand Dechoux
+
Matouk IFTISSEN 2013-07-04, 15:20
+
Nitin Pawar 2013-07-04, 15:30
+
Edward Capriolo 2013-07-04, 16:10
+
Raj Hadoop 2013-07-04, 16:17
+
Raj Hadoop 2013-07-04, 16:19
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB