Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Possibility of defining the Output directory programmatically


+
Manisha Gayathri 2012-07-23, 08:08
+
Vinod Singh 2012-07-23, 08:17
+
Manisha Gayathri 2012-07-23, 08:34
+
Vinod Singh 2012-07-23, 08:38
+
Manisha Gayathri 2012-07-23, 09:30
+
Vinod Singh 2012-07-23, 09:34
+
Manisha Gayathri 2012-07-23, 09:57
Copy link to this message
-
Re: Possibility of defining the Output directory programmatically
manisha,

set is variable key value pair, it does not execute any functions

if you want to use it like that then I would recomment write a shell
script which will generate all required things to a file and then
execute that file with hive -f  option

On Mon, Jul 23, 2012 at 3:27 PM, Manisha Gayathri <[EMAIL PROTECTED]> wrote:
> Thanks Vinod.
>
> I tried concatenating variables. But that is also not possible as I see.
> set pqr = concat(foo,bar);
> set file_name= home/user/Desktop
>
> Then the file_name I am getting is NOT  home/user/Desktop/foo_bar
> But what I am getting is, /home/user/Desktop/concat(foo,bar)
>
>
>
> On Mon, Jul 23, 2012 at 3:04 PM, Vinod Singh <[EMAIL PROTECTED]> wrote:
>>
>> We generate variables dynamically and then create a final script file by
>> concatenating variables (SET commands) and Hive queries. Then final script
>> is executed. Probably you can adopt something similar approach.
>>
>> Thanks,
>> Vinod
>>
>> 2012/7/23 Manisha Gayathri <[EMAIL PROTECTED]>
>>>
>>> Thanks again Vinod.
>>>
>>> Will try to find a way to pass the directory URLs from outside then.
>>> And it would be grateful if you can direct me to any guide or
>>> documentation that describes how to pass values from outside of Hive.
>>>
>>> Thanks
>>> Manisha
>>>
>>>
>>> On Mon, Jul 23, 2012 at 2:08 PM, Vinod Singh <[EMAIL PROTECTED]>
>>> wrote:
>>>>
>>>> SET commands are handled differently and UDFs can't be invoked there.
>>>> IMO you need to pass the directory location value from outside of Hive. That
>>>> is how we do.
>>>>
>>>> Thanks,
>>>> Vinod
>>>>
>>>> 2012/7/23 Manisha Gayathri <[EMAIL PROTECTED]>
>>>>>
>>>>> Hi Vinod,
>>>>>
>>>>> Thanks for the prompt reply.
>>>>> Understood your point and sorry for not providing the complete code
>>>>> segment earlier.
>>>>>
>>>>> I have the getFilePath function which should return a URL like this.
>>>>>
>>>>> home/user/Desktop/logDir/logs/log_0_testServer_2012_07_22
>>>>>
>>>>> The defined function works perfectly if I put getFilePath(
>>>>> "0","testServer" ) into the select statement.
>>>>>
>>>>> But I want to get that particular URL as the local directory name. (The
>>>>> requirement is such that this should not be hard-coded in the hive query.
>>>>> Rather should be generated in the custom UDF. )
>>>>>
>>>>> So can I do something like I v shown below?
>>>>>
>>>>> set file_name= getFilePath( "0","testServer" );    //Define a
>>>>> parameter.
>>>>> .................
>>>>> ..............
>>>>> INSERT OVERWRITE LOCAL DIRECTORY 'file:///${hiveconf:file_name}'
>>>>> //Assign the above parameter as the file URL
>>>>>
>>>>> I tried this way. But the directory name is returned as
>>>>>
>>>>> file:/getFilePath( "0" , "testServer" )
>>>>>
>>>>> Does that mean I cannot use UDF to define the local directory name? Or
>>>>> am I doing anything wrong in here?
>>>>>
>>>>> Thanks
>>>>> Manisha
>>>>>
>>>>> On Mon, Jul 23, 2012 at 1:47 PM, Vinod Singh <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>>
>>>>>> The output path in this query is already parameterized-
>>>>>>
>>>>>> INSERT OVERWRITE LOCAL DIRECTORY 'file:///${hiveconf:file_name}'
>>>>>>
>>>>>> Though UDF is not going to be invoked here.
>>>>>>
>>>>>> Thanks,
>>>>>> Vinod
>>>>>>
>>>>>>
>>>>>> 2012/7/23 Manisha Gayathri <[EMAIL PROTECTED]>
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> Is there any possibility of defining the output directory of a hive
>>>>>>> query using a Hive UDF?
>>>>>>>
>>>>>>> In my UDF, I am passing 2 parameters (as follows) and this generates
>>>>>>> a file-system URL
>>>>>>> getFilePath( "0","testServer" );
>>>>>>>
>>>>>>> Can I use the above getFilePath( "0","testServer" ) value, as the
>>>>>>> Local Directory location as follows?
>>>>>>> INSERT OVERWRITE LOCAL DIRECTORY 'file:///${hiveconf:file_name}'
>>>>>>>
>>>>>>> If above is not the correct way, could you please show me how to do
>>>>>>> it?
>>>>>>> Your help is much appreciated!
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> ~Regards
>>>>>>> Manisha Eleperuma
>>>>>>> Software Engineer
>>>>
Nitin Pawar
+
Vinod Singh 2012-07-23, 10:03
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB