Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Possibility of defining the Output directory programmatically


Copy link to this message
-
Re: Possibility of defining the Output directory programmatically
manisha,

set is variable key value pair, it does not execute any functions

if you want to use it like that then I would recomment write a shell
script which will generate all required things to a file and then
execute that file with hive -f  option

On Mon, Jul 23, 2012 at 3:27 PM, Manisha Gayathri <[EMAIL PROTECTED]> wrote:
> Thanks Vinod.
>
> I tried concatenating variables. But that is also not possible as I see.
> set pqr = concat(foo,bar);
> set file_name= home/user/Desktop
>
> Then the file_name I am getting is NOT  home/user/Desktop/foo_bar
> But what I am getting is, /home/user/Desktop/concat(foo,bar)
>
>
>
> On Mon, Jul 23, 2012 at 3:04 PM, Vinod Singh <[EMAIL PROTECTED]> wrote:
>>
>> We generate variables dynamically and then create a final script file by
>> concatenating variables (SET commands) and Hive queries. Then final script
>> is executed. Probably you can adopt something similar approach.
>>
>> Thanks,
>> Vinod
>>
>> 2012/7/23 Manisha Gayathri <[EMAIL PROTECTED]>
>>>
>>> Thanks again Vinod.
>>>
>>> Will try to find a way to pass the directory URLs from outside then.
>>> And it would be grateful if you can direct me to any guide or
>>> documentation that describes how to pass values from outside of Hive.
>>>
>>> Thanks
>>> Manisha
>>>
>>>
>>> On Mon, Jul 23, 2012 at 2:08 PM, Vinod Singh <[EMAIL PROTECTED]>
>>> wrote:
>>>>
>>>> SET commands are handled differently and UDFs can't be invoked there.
>>>> IMO you need to pass the directory location value from outside of Hive. That
>>>> is how we do.
>>>>
>>>> Thanks,
>>>> Vinod
>>>>
>>>> 2012/7/23 Manisha Gayathri <[EMAIL PROTECTED]>
>>>>>
>>>>> Hi Vinod,
>>>>>
>>>>> Thanks for the prompt reply.
>>>>> Understood your point and sorry for not providing the complete code
>>>>> segment earlier.
>>>>>
>>>>> I have the getFilePath function which should return a URL like this.
>>>>>
>>>>> home/user/Desktop/logDir/logs/log_0_testServer_2012_07_22
>>>>>
>>>>> The defined function works perfectly if I put getFilePath(
>>>>> "0","testServer" ) into the select statement.
>>>>>
>>>>> But I want to get that particular URL as the local directory name. (The
>>>>> requirement is such that this should not be hard-coded in the hive query.
>>>>> Rather should be generated in the custom UDF. )
>>>>>
>>>>> So can I do something like I v shown below?
>>>>>
>>>>> set file_name= getFilePath( "0","testServer" );    //Define a
>>>>> parameter.
>>>>> .................
>>>>> ..............
>>>>> INSERT OVERWRITE LOCAL DIRECTORY 'file:///${hiveconf:file_name}'
>>>>> //Assign the above parameter as the file URL
>>>>>
>>>>> I tried this way. But the directory name is returned as
>>>>>
>>>>> file:/getFilePath( "0" , "testServer" )
>>>>>
>>>>> Does that mean I cannot use UDF to define the local directory name? Or
>>>>> am I doing anything wrong in here?
>>>>>
>>>>> Thanks
>>>>> Manisha
>>>>>
>>>>> On Mon, Jul 23, 2012 at 1:47 PM, Vinod Singh <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>>
>>>>>> The output path in this query is already parameterized-
>>>>>>
>>>>>> INSERT OVERWRITE LOCAL DIRECTORY 'file:///${hiveconf:file_name}'
>>>>>>
>>>>>> Though UDF is not going to be invoked here.
>>>>>>
>>>>>> Thanks,
>>>>>> Vinod
>>>>>>
>>>>>>
>>>>>> 2012/7/23 Manisha Gayathri <[EMAIL PROTECTED]>
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> Is there any possibility of defining the output directory of a hive
>>>>>>> query using a Hive UDF?
>>>>>>>
>>>>>>> In my UDF, I am passing 2 parameters (as follows) and this generates
>>>>>>> a file-system URL
>>>>>>> getFilePath( "0","testServer" );
>>>>>>>
>>>>>>> Can I use the above getFilePath( "0","testServer" ) value, as the
>>>>>>> Local Directory location as follows?
>>>>>>> INSERT OVERWRITE LOCAL DIRECTORY 'file:///${hiveconf:file_name}'
>>>>>>>
>>>>>>> If above is not the correct way, could you please show me how to do
>>>>>>> it?
>>>>>>> Your help is much appreciated!
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> ~Regards
>>>>>>> Manisha Eleperuma
>>>>>>> Software Engineer
>>>>
Nitin Pawar