Hi

I am using Hive with Microsoft PowerPivot as the visualization tool.

When I am running a query involving UDF like this from PowerPivot:
add jar /usr/local/hadoop/src/retweetlink1.jar;
create temporary function link as 'retweetlink';
Followed by a select statement, the query executes fine for the 1st and 2nd
round of map-reduce jobs, but when the session changes then throws an error
like retweetlink class is not found.

How do I make an UDF support across multiple sessions.

Thanks
Sreenath Mullassery

include it in your ~/.hiverc to have it across sessions

On Mon, Jun 11, 2012 at 5:42 PM, Sreenath Menon <[EMAIL PROTECTED]>wrote:

> Hi
>
> I am using Hive with Microsoft PowerPivot as the visualization tool.
>
> When I am running a query involving UDF like this from PowerPivot:
> add jar /usr/local/hadoop/src/retweetlink1.jar;
> create temporary function link as 'retweetlink';
> Followed by a select statement, the query executes fine for the 1st and
> 2nd round of map-reduce jobs, but when the session changes then throws an
> error like retweetlink class is not found.
>
> How do I make an UDF support across multiple sessions.
>
> Thanks
> Sreenath Mullassery
>

--
Nitin Pawar

Hi Nitin

Can u kindly help me (briefly) on how to add to hiverc...no such location
exsist in my machine

in your home directory (if you are using linux with vm) then you will need
to create that file and add the entries exactly the same way you add in
hive cli

On Mon, Jun 11, 2012 at 6:06 PM, Sreenath Menon <[EMAIL PROTECTED]>wrote:

> Hi Nitin
>
> Can u kindly help me (briefly) on how to add to hiverc...no such location
> exsist in my machine
>

--
Nitin Pawar

K..so i have created a file 'sample.hiverc' in the home directory..how do I
run this particular file

if you have created a file other than name ".hiverc" , you will need to
start hive with this file

something like hive -i hiverc

but when you create a  file .hiverc in your home directory hive cli picks
it up automatically

On Mon, Jun 11, 2012 at 6:13 PM, Sreenath Menon <[EMAIL PROTECTED]>wrote:

> K..so i have created a file 'sample.hiverc' in the home directory..how do
> I run this particular file
>

--
Nitin Pawar

This works when I am using Hive cli.
i.e. When I give: select link(tweet) from tweetsdata; in the cli and
defined the function 'link' in .hiverc

But when i run /usr/hive --service hiveserver
And use the function in PowerPivot, then it says that 'link' is not defined.

Nithin,

Any idea on invoking .hiverc when running : /usr/hive/bin/hive --service
hiveserver

I have tried that before. It does not work. But anyways thanks for the
suggestion.
Anyone else who can be of help in this area?

UDF's do not live across session. This is why the syntax is "CREATE
TEMPORARY FUNCTION". You can build the function into hive and then you
will not need to add the UDF.
On Mon, Jun 11, 2012 at 11:31 AM, Sreenath Menon
<[EMAIL PROTECTED]> wrote:
> I have tried that before. It does not work. But anyways thanks for the
> suggestion.
> Anyone else who can be of help in this area?

Ya UDF do not live across section. But what if I just want the temporary
function to be created each time of a new session. This is what is done
with help of .hiverc. But again this is working only with cli mode not in
server mode.

BTW I am interested to know how to build the function into hive, kindly
guide me on how to move forward in this, Edward.

you can checkout hive code and build your udf and ship it with hive :)

custom hive for yourself ... if the function is generic feel free to share
on git :)

On Mon, Jun 11, 2012 at 9:28 PM, Sreenath Menon <[EMAIL PROTECTED]>wrote:

> Ya UDF do not live across section. But what if I just want the temporary
> function to be created each time of a new session. This is what is done
> with help of .hiverc. But again this is working only with cli mode not in
> server mode.
>
> BTW I am interested to know how to build the function into hive, kindly
> guide me on how to move forward in this, Edward.
>

--
Nitin Pawar

I have a jar file : 'twittergen.jar', now how can I add it to hive lib.
Kindly help. I need the function to be used across sections when running a
server instance. Now stuck up with this.

put jar in hive-classpath (libs directory etc) and do a create temporary
function every time you connect from server.

What version of hive are you on?

~Aniket

On Mon, Jun 11, 2012 at 11:12 PM, Sreenath Menon
<[EMAIL PROTECTED]>wrote:

> I have a jar file : 'twittergen.jar', now how can I add it to hive lib.
> Kindly help. I need the function to be used across sections when running a
> server instance. Now stuck up with this.
>

--
"...:::Aniket:::... Quetzalco@tl"

I mean every time you connect to hive server-

execute-
create temporary function...;
your hive query...;

~Aniket

On Mon, Jun 11, 2012 at 11:27 PM, Aniket Mokashi <[EMAIL PROTECTED]>wrote:

> put jar in hive-classpath (libs directory etc) and do a create temporary
> function every time you connect from server.
>
> What version of hive are you on?
>
> ~Aniket
>
>
> On Mon, Jun 11, 2012 at 11:12 PM, Sreenath Menon <[EMAIL PROTECTED]
> > wrote:
>
>> I have a jar file : 'twittergen.jar', now how can I add it to hive lib.
>> Kindly help. I need the function to be used across sections when running
>> a server instance. Now stuck up with this.
>>
>
>
>
> --
> "...:::Aniket:::... Quetzalco@tl"
>

--
"...:::Aniket:::... Quetzalco@tl"

aniket, his problem is that he does not want to create that function each
time.

he wants it available on each session with hive server
so we are suggesting custom hive build where he will bundle his udf with
hive and have available with hive server

On Tue, Jun 12, 2012 at 11:58 AM, Aniket Mokashi <[EMAIL PROTECTED]>wrote:

> I mean every time you connect to hive server-
>
> execute-
> create temporary function...;
> your hive query...;
>
> ~Aniket
>
> On Mon, Jun 11, 2012 at 11:27 PM, Aniket Mokashi <[EMAIL PROTECTED]>wrote:
>
>> put jar in hive-classpath (libs directory etc) and do a create temporary
>> function every time you connect from server.
>>
>> What version of hive are you on?
>>
>> ~Aniket
>>
>>
>> On Mon, Jun 11, 2012 at 11:12 PM, Sreenath Menon <
>> [EMAIL PROTECTED]> wrote:
>>
>>> I have a jar file : 'twittergen.jar', now how can I add it to hive lib.
>>> Kindly help. I need the function to be used across sections when running
>>> a server instance. Now stuck up with this.
>>>
>>
>>
>>
>> --
>> "...:::Aniket:::... Quetzalco@tl"
>>
>
>
>
> --
> "...:::Aniket:::... Quetzalco@tl"
>

--
Nitin Pawar

Any other way Nitin, I just want to add a single jar file and do not know
much about custom hive build. And this requirement may vary at some other
point of time. Its not a good way of building hive each time I need a new
jar to be added.

Like is there anyway to make the .hiverc file be executed even in
hiveserver instance.
simple way like this
hive --service hiveserver -i .hiverc
doesnot work Nithin

I got that -i option is not applicable to hive server

let me drill through to find out if there is any other option

on the other side to be curious , if you are writing the code to access
hiveserver, whats the harm defining these parameters in your code?

On Tue, Jun 12, 2012 at 12:12 PM, Sreenath Menon
<[EMAIL PROTECTED]>wrote:

> Like is there anyway to make the .hiverc file be executed even in
> hiveserver instance.
> simple way like this
> hive --service hiveserver -i .hiverc
> doesnot work Nithin
>

--
Nitin Pawar

Yes understood. I do not have a problem in defining the parameters in the
code. But the problem is, I am using PowerPivot as the visualization engine.

Now, when I give the query as a set like:
--------------------

add jar /usr/local/hadoop/src/retweetlink1.jar;

create temporary function link as 'retweetlink';

select link(tweet),count(*) as countlink from tweetsdata where tweet like
'%RT%' group by link(tweet) sort by countlink desc limit 20;

--------------------

The error then is that /usr/local/hadoop/src/retweetlink1.jar; does not
exist

So, I need to give them as seperate queries like add jar first, execute
that followed by others.

Now, the problem is it runs fine for the first instance of 3 mapreduce
jobs, but when the second instance runs, for importing data, then throws up
error saying that retweetlink class not found.

There are multiple instances of 3 mpareduce jobs (executing one after the
other) on running the single query using powerpivot.
I can find out next instance when this throws up in the screen after like
2 instance of the 3mapreduce jobs.
Hive history
file=/tmp/hadoop/hive_job_log_hadoop_201206121120_1912521929.txt
Understood nitin?

By instance I mean a set of mapreduce jobs (3 in this case)......when
executing in Cli only one instance runs and output is displayed on the
screen, but this is not the case when using with PowerPivot (multiple
instance one after the other and contains the same no. of HDFS read
write...etc!!!!)...also would like to know y?