Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - How Can I store the Hive query result in one file ?


Copy link to this message
-
Re: How Can I store the Hive query result in one file ?
Matouk IFTISSEN 2013-07-04, 15:20
Thanks for your responses,
effctively  the answer of Bertrand make this possible: the set of hive
properities below froce thet job to write the hive result in one file
whithout specifing the name (0000_0) :
 set hive.exec.reducers.max = 1;

set mapred.reduce.tasks = 1;

for Nitin, I want to store the results of SELECT not the stdout (log) of
execution of the query, is this applicable for the results of SELECT?
2013/7/4 Michael Malak <[EMAIL PROTECTED]>

> I have found that for output larger than a few GB, redirecting stdout
> results in an incomplete file.  For very large output, I do CREATE TABLE
> MYTABLE AS SELECT ... and then copy the resulting HDFS files directly out
> of /user/hive/warehouse.
>
>    *From:* Bertrand Dechoux <[EMAIL PROTECTED]>
> *To:* [EMAIL PROTECTED]
> *Sent:* Thursday, July 4, 2013 7:09 AM
> *Subject:* Re: How Can I store the Hive query result in one file ?
>
> The question is what is the volume of your output. There is one file per
> output task (map or reduce) because that way each can write it
> independently and in parallel. That's how mapreduce work. And except by
> forcing the number of tasks to 1, there is no certain way to have one
> output file.
>
> But indeed if the volume is low enough, you could also capture the
> standard output into a local file like Nitin described.
>
> Bertrand
>
>
> On Thu, Jul 4, 2013 at 12:38 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:
>
> will hive -e "query" > filename  or hive -f query.q > filename will do ?
>
> you specially want it to write into a named file on hdfs only?
>
>
> On Thu, Jul 4, 2013 at 3:12 PM, Matouk IFTISSEN <
> [EMAIL PROTECTED]> wrote:
>
> Hello Hive users,
> Is there a manner to store the Hive  query result (SELECT *.....) in a
> specfique and alone file (given the file name) like (INSERT OVERWRITE LOCAL
> DIRECTORY '/directory_path_name/')?
> Thanks for your answers
>
>
>
>
> --
> Nitin Pawar
>
>
>
>
> --
> Bertrand Dechoux
>
>
>