Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Rename an output file in hive {was: Re: Possibility of defining the Output directory programmatically}


+
Manisha Gayathri 2012-07-27, 06:26
Copy link to this message
-
Re: Rename an output file in hive {was: Re: Possibility of defining the Output directory programmatically}
Hi Manisha

In mapreduce if you want to change the name of output file you may need to write your own OutputFormat.

Renaming files in hdfs is straight forward
hadoop fs -mv oldFileName newFileName
Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: Manisha Gayathri <[EMAIL PROTECTED]>
Date: Fri, 27 Jul 2012 11:56:57
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Rename an output file in hive {was: Re: Possibility of defining the
 Output directory programmatically}

Hi all,

Thanks alot for your help.
With your advice, could successfully pass the needed parameters and
generate the output file as required.

In my use case, the output file needed to be a .gz file. When I run my
query with the following commands, the file that gets generated is named as
*000000_0.gz*
set mapred.output.compress=true;
set hive.exec.compress.output=true;
set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
set io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;
Is there anyway that I can rename this file or generate the file in a name
of my choice from hive it self?

Thanks again.
Manisha

On Mon, Jul 23, 2012 at 3:35 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:

> manisha,
>
> set is variable key value pair, it does not execute any functions
>
> if you want to use it like that then I would recomment write a shell
> script which will generate all required things to a file and then
> execute that file with hive -f  option
>
> On Mon, Jul 23, 2012 at 3:27 PM, Manisha Gayathri <[EMAIL PROTECTED]>
> wrote:
> > Thanks Vinod.
> >
> > I tried concatenating variables. But that is also not possible as I see.
> > set pqr = concat(foo,bar);
> > set file_name= home/user/Desktop
> >
> > Then the file_name I am getting is NOT  home/user/Desktop/foo_bar
> > But what I am getting is, /home/user/Desktop/concat(foo,bar)
> >
> >
> >
> > On Mon, Jul 23, 2012 at 3:04 PM, Vinod Singh <[EMAIL PROTECTED]>
> wrote:
> >>
> >> We generate variables dynamically and then create a final script file by
> >> concatenating variables (SET commands) and Hive queries. Then final
> script
> >> is executed. Probably you can adopt something similar approach.
> >>
> >> Thanks,
> >> Vinod
> >>
> >> 2012/7/23 Manisha Gayathri <[EMAIL PROTECTED]>
> >>>
> >>> Thanks again Vinod.
> >>>
> >>> Will try to find a way to pass the directory URLs from outside then.
> >>> And it would be grateful if you can direct me to any guide or
> >>> documentation that describes how to pass values from outside of Hive.
> >>>
> >>> Thanks
> >>> Manisha
> >>>
> >>>
> >>> On Mon, Jul 23, 2012 at 2:08 PM, Vinod Singh <[EMAIL PROTECTED]>
> >>> wrote:
> >>>>
> >>>> SET commands are handled differently and UDFs can't be invoked there.
> >>>> IMO you need to pass the directory location value from outside of
> Hive. That
> >>>> is how we do.
> >>>>
> >>>> Thanks,
> >>>> Vinod
> >>>>
> >>>> 2012/7/23 Manisha Gayathri <[EMAIL PROTECTED]>
> >>>>>
> >>>>> Hi Vinod,
> >>>>>
> >>>>> Thanks for the prompt reply.
> >>>>> Understood your point and sorry for not providing the complete code
> >>>>> segment earlier.
> >>>>>
> >>>>> I have the getFilePath function which should return a URL like this.
> >>>>>
> >>>>> home/user/Desktop/logDir/logs/log_0_testServer_2012_07_22
> >>>>>
> >>>>> The defined function works perfectly if I put getFilePath(
> >>>>> "0","testServer" ) into the select statement.
> >>>>>
> >>>>> But I want to get that particular URL as the local directory name.
> (The
> >>>>> requirement is such that this should not be hard-coded in the hive
> query.
> >>>>> Rather should be generated in the custom UDF. )
> >>>>>
> >>>>> So can I do something like I v shown below?
> >>>>>
> >>>>> set file_name= getFilePath( "0","testServer" );    //Define a
> >>>>> parameter.
> >>>>> .................
> >>>>> ..............
> >>>>> INSERT OVERWRITE LOCAL DIRECTORY 'file:///${hiveconf:file_name}'
> >>>>> //Assign the above parameter as the file URL

~Regards
*Manisha Eleperuma*
Software Engineer
WSO2, Inc.: http://wso2.com
lean.enterprise.middleware

*
*
*
*
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB