Hi guys,

I found that HiveServer can fetch metadata thourgh Hive class after
generated a task like DDLTask, and meanwhile Hive class is a client
who ask HiveServer for metadata. So I guess it may be possible
hiveserver both be a server and a client of itself, am I right?
Thanks,
Min
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Is there anyone can answer my question?
Thanks in advanced!

On Sun, May 10, 2009 at 12:00 AM, Min Zhou <[EMAIL PROTECTED]> wrote:

> Hi guys,
>
> I found that HiveServer can fetch metadata thourgh Hive class after
> generated a task like DDLTask, and meanwhile Hive class is a client
> who ask HiveServer for metadata. So I guess it may be possible
> hiveserver both be a server and a client of itself, am I right?
>
>
> Thanks,
> Min
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>

--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Apologies for the delay Min..

The HiveServer code calls the metastore thrift api to get the metadata. The Hive.class in ql also talks to the metastore api to get the metadata. Not sure if I follow your question though?

Ashish

________________________________
From: Min Zhou [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, May 13, 2009 6:07 PM
To: hive-user
Subject: Re: Is it possible hiveserver both be a server and a client of itself?

Is there anyone can answer my question?
Thanks in advanced!

On Sun, May 10, 2009 at 12:00 AM, Min Zhou <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi guys,

I found that HiveServer can fetch metadata thourgh Hive class after
generated a task like DDLTask, and meanwhile Hive class is a client
who ask HiveServer for metadata. So I guess it may be possible
hiveserver both be a server and a client of itself, am I right?
Thanks,
Min
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Hi  Ashish,
Thank you for your swift reply. I guess it's HiveServer code inherit the
metastore thrift api. HiveServer provide metadata service by itself,  not
call another metastore server,  and Hive.class talks to the HiveServer to
get the metadata.
Regards,
Min

On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <[EMAIL PROTECTED]> wrote:

>  Apologies for the delay Min..
>
> The HiveServer code calls the metastore thrift api to get the metadata. The
> Hive.class in ql also talks to the metastore api to get the metadata. Not
> sure if I follow your question though?
>
> Ashish
>
>  ------------------------------
> *From:* Min Zhou [mailto:[EMAIL PROTECTED]]
> *Sent:* Wednesday, May 13, 2009 6:07 PM
> *To:* hive-user
> *Subject:* Re: Is it possible hiveserver both be a server and a client of
> itself?
>
> Is there anyone can answer my question?
> Thanks in advanced!
>
> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <[EMAIL PROTECTED]> wrote:
>
>> Hi guys,
>>
>> I found that HiveServer can fetch metadata thourgh Hive class after
>> generated a task like DDLTask, and meanwhile Hive class is a client
>> who ask HiveServer for metadata. So I guess it may be possible
>> hiveserver both be a server and a client of itself, am I right?
>>
>>
>> Thanks,
>> Min
>> --
>> My research interests are distributed systems, parallel computing and
>> bytecode based virtual machine.
>>
>> My profile:
>> http://www.linkedin.com/in/coderplay
>> My blog:
>> http://coderplay.javaeye.com
>>
>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>

--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

I don't think that would work, where would you store the meta-data for
the meta-data?

Min Zhou wrote:
> Hi  Ashish,
> Thank you for your swift reply. I guess it's HiveServer code inherit
> the metastore thrift api. HiveServer provide metadata service by
> itself,  not call another metastore server,  and Hive.class talks to
> the HiveServer to get the metadata.
> Regards,
> Min
>
> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     Apologies for the delay Min..
>      
>     The HiveServer code calls the metastore thrift api to get the
>     metadata. The Hive.class in ql also talks to the metastore api to
>     get the metadata. Not sure if I follow your question though?
>      
>     Ashish
>
>     ------------------------------------------------------------------------
>     *From:* Min Zhou [mailto:[EMAIL PROTECTED]
>     <mailto:[EMAIL PROTECTED]>]
>     *Sent:* Wednesday, May 13, 2009 6:07 PM
>     *To:* hive-user
>     *Subject:* Re: Is it possible hiveserver both be a server and a
>     client of itself?
>
>     Is there anyone can answer my question?
>     Thanks in advanced!
>
>     On Sun, May 10, 2009 at 12:00 AM, Min Zhou <[EMAIL PROTECTED]
>     <mailto:[EMAIL PROTECTED]>> wrote:
>
>         Hi guys,
>
>         I found that HiveServer can fetch metadata thourgh Hive class
>         after
>         generated a task like DDLTask, and meanwhile Hive class is a
>         client
>         who ask HiveServer for metadata. So I guess it may be possible
>         hiveserver both be a server and a client of itself, am I right?
>
>
>         Thanks,
>         Min
>         --
>         My research interests are distributed systems, parallel
>         computing and
>         bytecode based virtual machine.
>
>         My profile:
>         http://www.linkedin.com/in/coderplay
>         My blog:
>         http://coderplay.javaeye.com
>
>
>
>
>     --
>     My research interests are distributed systems, parallel computing
>     and bytecode based virtual machine.
>
>     My profile:
>     http://www.linkedin.com/in/coderplay
>     My blog:
>     http://coderplay.javaeye.com
>
>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com

Sorry, I didn't precisely express my meaning.  I didn't talk about a
supposition, but talked about a possible phenomenon may happen at hive.
The hiveserver may both be a server and a client of itself,  you can look at
the definition of HiveServer

public static class HiveServerHandler extends HiveMetaStore.HMSHandler ... {
   ...
}

It extends HiveMetaStore.HMSHandler, which provide the same function of
metastore .

public class HiveMetaStore extends ThriftHiveMetastore {

    public static class HMSHandler extends FacebookBase implements
ThriftHiveMetastore.Iface {
     // omit
    }

    // main entry of the MetaStore but didn't be called at this case
    public static void main(String[] args) {
    ...
    }
}

there is no another standalone metastore server if you have up a hiveserver,
which is also a metaserver itself.

Min
On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <[EMAIL PROTECTED]> wrote:

>
> I don't think that would work, where would you store the meta-data for the
> meta-data?
>
>
> Min Zhou wrote:
>
> Hi  Ashish,
> Thank you for your swift reply. I guess it's HiveServer code inherit the
> metastore thrift api. HiveServer provide metadata service by itself,  not
> call another metastore server,  and Hive.class talks to the HiveServer to
> get the metadata.
> Regards,
> Min
>
> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <[EMAIL PROTECTED]>wrote:
>
>>  Apologies for the delay Min..
>>
>> The HiveServer code calls the metastore thrift api to get the metadata.
>> The Hive.class in ql also talks to the metastore api to get the metadata.
>> Not sure if I follow your question though?
>>
>> Ashish
>>
>>  ------------------------------
>> *From:* Min Zhou [mailto:[EMAIL PROTECTED]]
>> *Sent:* Wednesday, May 13, 2009 6:07 PM
>> *To:* hive-user
>> *Subject:* Re: Is it possible hiveserver both be a server and a client of
>> itself?
>>
>>  Is there anyone can answer my question?
>> Thanks in advanced!
>>
>> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <[EMAIL PROTECTED]> wrote:
>>
>>> Hi guys,
>>>
>>> I found that HiveServer can fetch metadata thourgh Hive class after
>>> generated a task like DDLTask, and meanwhile Hive class is a client
>>> who ask HiveServer for metadata. So I guess it may be possible
>>> hiveserver both be a server and a client of itself, am I right?
>>>
>>>
>>> Thanks,
>>> Min
>>> --
>>> My research interests are distributed systems, parallel computing and
>>> bytecode based virtual machine.
>>>
>>> My profile:
>>> http://www.linkedin.com/in/coderplay
>>> My blog:
>>> http://coderplay.javaeye.com
>>>
>>
>>
>>
>> --
>> My research interests are distributed systems, parallel computing and
>> bytecode based virtual machine.
>>
>> My profile:
>> http://www.linkedin.com/in/coderplay
>> My blog:
>> http://coderplay.javaeye.com
>>
>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>
>
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

We have used thrift's inheritance feature where one service can extend
another service, i.e., HiveServer extends HiveMetaStore. This means that
1. HiveServer exposes HiveMetaStore methods
2. HiveClient can be used to invoke methods exposed by both HiveServer and
HiveMetaStore.

Does that make sense?
On 5/13/09 7:52 PM, "Min Zhou" <[EMAIL PROTECTED]> wrote:

> Sorry, I didn't precisely express my meaning.  I didn't talk about a
> supposition, but talked about a possible phenomenon may happen at hive.
> The hiveserver may both be a server and a client of itself,  you can look at
> the definition of HiveServer
>
> public static class HiveServerHandler extends HiveMetaStore.HMSHandler ... {
>    ...
> }
>
>
>
> It extends HiveMetaStore.HMSHandler, which provide the same function of
> metastore .
>
> public class HiveMetaStore extends ThriftHiveMetastore {
>  
>     public static class HMSHandler extends FacebookBase implements
> ThriftHiveMetastore.Iface {
>      // omit
>     }
>    
>     // main entry of the MetaStore but didn't be called at this case
>     public static void main(String[] args) {
>     ...
>     }
> }
>
>  there is no another standalone metastore server if you have up a hiveserver,
> which is also a metaserver itself.
>
> Min
> On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <[EMAIL PROTECTED]> wrote:
>>  
>>
>> I don't think that would work, where would you store the meta-data for the
>> meta-data?
>>
>>
>> Min Zhou wrote:
>>> Hi  Ashish,
>>> Thank you for your swift reply. I guess it's HiveServer code inherit the
>>> metastore thrift api. HiveServer provide metadata service by itself,  not
>>> call another metastore server,  and Hive.class talks to the HiveServer to
>>> get
>>> the metadata.
>>> Regards,
>>> Min
>>>  
>>>  
>>> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <[EMAIL PROTECTED]> wrote:
>>>  
>>>>  
>>>>  
>>>> Apologies for the delay Min..
>>>>  
>>>>  
>>>>  
>>>> The HiveServer code calls the metastore thrift api to get the metadata. The
>>>> Hive.class in ql also talks to the metastore api to get the metadata. Not
>>>> sure if I follow your question though?
>>>>  
>>>>  
>>>>  
>>>> Ashish
>>>>  
>>>>  
>>>>  
>>>>
>>>> From: Min Zhou [mailto:[EMAIL PROTECTED]]
>>>>  Sent: Wednesday, May 13, 2009 6:07 PM
>>>>  To: hive-user
>>>>  Subject: Re: Is it possible hiveserver both be a server and a client of
>>>> itself?
>>>>  
>>>>  
>>>>  
>>>>  
>>>> Is there anyone can answer my question? 
>>>> Thanks in advanced!
>>>>  
>>>>  
>>>> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <[EMAIL PROTECTED]> wrote:
>>>>  
>>>>> Hi guys,
>>>>>  
>>>>> I found that HiveServer can fetch metadata thourgh Hive class after
>>>>> generated a task like DDLTask, and meanwhile Hive class is a client
>>>>> who ask HiveServer for metadata. So I guess it may be possible
>>>>> hiveserver both be a server and a client of itself, am I right?
>>>>>  
>>>>>  
>>>>> Thanks,
>>>>> Min
>>>>>  --
>>>>> My research interests are distributed systems, parallel computing and
>>>>> bytecode based virtual machine.
>>>>>  
>>>>> My profile:
>>>>>  http://www.linkedin.com/in/coderplay
>>>>> My blog:
>>>>>  http://coderplay.javaeye.com
>>>>>  
>>>>  
>>>>  
>>>>  
>>>>  
>>>> --
>>>> My research interests are distributed systems, parallel computing and
>>>> bytecode based virtual machine.
>>>>  
>>>> My profile:
>>>>  http://www.linkedin.com/in/coderplay
>>>> My blog:
>>>>  http://coderplay.javaeye.com
>>>>  
>>>>  
>>>>  
>>>>  
>>>  
>>>  
>>>  
>>>  
>>> --
>>> My research interests are distributed systems, parallel computing and
>>> bytecode based virtual machine.
>>>  
>>> My profile:
>>>  http://www.linkedin.com/in/coderplay
>>> My blog:
>>>  http://coderplay.javaeye.com
>
>

Hey Raghu,

Agreed with you. My question is ,  Hive.java code called by HiveServer,  and
Hive.java would talked to HiveServer through a socket connection. Am I
right?

Min
On Thu, May 14, 2009 at 11:02 AM, Raghu Murthy <[EMAIL PROTECTED]> wrote:

> We have used thrift's inheritance feature where one service can extend
> another service, i.e., HiveServer extends HiveMetaStore. This means that
> 1. HiveServer exposes HiveMetaStore methods
> 2. HiveClient can be used to invoke methods exposed by both HiveServer and
> HiveMetaStore.
>
> Does that make sense?
>
>
> On 5/13/09 7:52 PM, "Min Zhou" <[EMAIL PROTECTED]> wrote:
>
> > Sorry, I didn't precisely express my meaning.  I didn't talk about a
> > supposition, but talked about a possible phenomenon may happen at hive.
> > The hiveserver may both be a server and a client of itself,  you can look
> at
> > the definition of HiveServer
> >
> > public static class HiveServerHandler extends HiveMetaStore.HMSHandler
> ... {
> >    ...
> > }
> >
> >
> >
> > It extends HiveMetaStore.HMSHandler, which provide the same function of
> > metastore .
> >
> > public class HiveMetaStore extends ThriftHiveMetastore {
> >
> >     public static class HMSHandler extends FacebookBase implements
> > ThriftHiveMetastore.Iface {
> >      // omit
> >     }
> >
> >     // main entry of the MetaStore but didn't be called at this case
> >     public static void main(String[] args) {
> >     ...
> >     }
> > }
> >
> >  there is no another standalone metastore server if you have up a
> hiveserver,
> > which is also a metaserver itself.
> >
> > Min
> > On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <[EMAIL PROTECTED]>
> wrote:
> >>
> >>
> >> I don't think that would work, where would you store the meta-data for
> the
> >> meta-data?
> >>
> >>
> >> Min Zhou wrote:
> >>> Hi  Ashish,
> >>> Thank you for your swift reply. I guess it's HiveServer code inherit
> the
> >>> metastore thrift api. HiveServer provide metadata service by itself,
> not
> >>> call another metastore server,  and Hive.class talks to the HiveServer
> to
> >>> get
> >>> the metadata.
> >>> Regards,
> >>> Min
> >>>
> >>>
> >>> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>>>
> >>>>
> >>>> Apologies for the delay Min..
> >>>>
> >>>>
> >>>>
> >>>> The HiveServer code calls the metastore thrift api to get the
> metadata. The
> >>>> Hive.class in ql also talks to the metastore api to get the metadata.
> Not
> >>>> sure if I follow your question though?
> >>>>
> >>>>
> >>>>
> >>>> Ashish
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> From: Min Zhou [mailto:[EMAIL PROTECTED]]
> >>>>  Sent: Wednesday, May 13, 2009 6:07 PM
> >>>>  To: hive-user
> >>>>  Subject: Re: Is it possible hiveserver both be a server and a client
> of
> >>>> itself?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Is there anyone can answer my question?
> >>>> Thanks in advanced!
> >>>>
> >>>>
> >>>> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <[EMAIL PROTECTED]>
> wrote:
> >>>>
> >>>>> Hi guys,
> >>>>>
> >>>>> I found that HiveServer can fetch metadata thourgh Hive class after
> >>>>> generated a task like DDLTask, and meanwhile Hive class is a client
> >>>>> who ask HiveServer for metadata. So I guess it may be possible
> >>>>> hiveserver both be a server and a client of itself, am I right?
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>> Min
> >>>>>  --
> >>>>> My research interests are distributed systems, parallel computing and
> >>>>> bytecode based virtual machine.
> >>>>>
> >>>>> My profile:
> >>>>>  http://www.linkedin.com/in/coderplay
> >>>>> My blog:
> >>>>>  http://coderplay.javaeye.com
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> My research interests are distributed systems, parallel computing and
> >>>> bytecode based virtual machine.
> >>>>
> >>>> My profile:
> >>>>  http://www.linkedin.com/in/coderplay
> >>>> My blog:
> >>>>  http://coderplay.javaeye.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> My research interests are distributed systems, parallel computing and
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Hive.java does not call methods in HiveServer. Can you explain what you are
trying to achieve? You should just be able to start the HiveServer and then
create a HiveClient to invoke methods of both HiveServer itself and
HiveMetaStore.

On 5/13/09 8:07 PM, "Min Zhou" <[EMAIL PROTECTED]> wrote:

> Hey Raghu,
>
> Agreed with you. My question is ,  Hive.java code called by HiveServer,  and
> Hive.java would talked to HiveServer through a socket connection. Am I right?
>
> Min
> On Thu, May 14, 2009 at 11:02 AM, Raghu Murthy <[EMAIL PROTECTED]> wrote:
>> We have used thrift's inheritance feature where one service can extend
>> another service, i.e., HiveServer extends HiveMetaStore. This means that
>> 1. HiveServer exposes HiveMetaStore methods
>> 2. HiveClient can be used to invoke methods exposed by both HiveServer and
>> HiveMetaStore.
>>
>> Does that make sense?
>>
>>
>> On 5/13/09 7:52 PM, "Min Zhou" <[EMAIL PROTECTED]> wrote:
>>
>>>> Sorry, I didn't precisely express my meaning.  I didn't talk about a
>>>> supposition, but talked about a possible phenomenon may happen at hive.
>>>> The hiveserver may both be a server and a client of itself,  you can look
>>>> >> at
>>>> the definition of HiveServer
>>>>
>>>> public static class HiveServerHandler extends HiveMetaStore.HMSHandler ...
>>>> {
>>>>    ...
>>>> }
>>>>
>>>>
>>>>
>>>> It extends HiveMetaStore.HMSHandler, which provide the same function of
>>>> metastore .
>>>>
>>>> public class HiveMetaStore extends ThriftHiveMetastore {
>>>>  
>>>>     public static class HMSHandler extends FacebookBase implements
>>>> ThriftHiveMetastore.Iface {
>>>>      // omit
>>>>     }
>>>>    
>>>>     // main entry of the MetaStore but didn't be called at this case
>>>>     public static void main(String[] args) {
>>>>     ...
>>>>     }
>>>> }
>>>>
>>>>  there is no another standalone metastore server if you have up a
>>> hiveserver,
>>>> which is also a metaserver itself.
>>>>
>>>> Min
>>>> On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <[EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>
>>>>>> I don't think that would work, where would you store the meta-data for
>>>>>> >>> the
>>>>>> meta-data?
>>>>>>
>>>>>>
>>>>>> Min Zhou wrote:
>>>>>>>> Hi  Ashish,
>>>>>>>> Thank you for your swift reply. I guess it's HiveServer code inherit
>>>>>>>> the
>>>>>>>> metastore thrift api. HiveServer provide metadata service by itself, 
>>>>>>>> not
>>>>>>>> call another metastore server,  and Hive.class talks to the HiveServer
>>>>>>>> to
>>>>>>>> get
>>>>>>>> the metadata.
>>>>>>>> Regards,
>>>>>>>> Min
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Apologies for the delay Min..
>>>>>>>>>>
>>>>>>>>>>  
>>>>>>>>>>
>>>>>>>>>> The HiveServer code calls the metastore thrift api to get the
>>>>>> metadata. The
>>>>>>>>>> Hive.class in ql also talks to the metastore api to get the metadata.
>>>>>>>>>> Not
>>>>>>>>>> sure if I follow your question though?
>>>>>>>>>>
>>>>>>>>>>  
>>>>>>>>>>
>>>>>>>>>> Ashish
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> From: Min Zhou [mailto:[EMAIL PROTECTED]]
>>>>>>>>>>  Sent: Wednesday, May 13, 2009 6:07 PM
>>>>>>>>>>  To: hive-user
>>>>>>>>>>  Subject: Re: Is it possible hiveserver both be a server and a client
>>>>>>>>>> of
>>>>>>>>>> itself?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Is there anyone can answer my question? 
>>>>>>>>>> Thanks in advanced!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, May 10, 2009 at 12:00 AM, Min Zhou <[EMAIL PROTECTED]>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>
>>>>>>>>>>>> I found that HiveServer can fetch metadata thourgh Hive class after
>>>>>>>>>>>> generated a task like DDLTask, and meanwhile Hive class is a client
>>>>>>>>>>>> who ask HiveServer for metadata. So I guess it may be possible
>>>>>>>>>>>> hiveserver both be a server and a client of itself, am I right?

HiveServer does call methods in Hive.java when compiling and planing a query
command.  HiveServer talk to itself through Hive.java and a socket
connection.
This is the logic of the current hive trunk code.

On Thu, May 14, 2009 at 11:16 AM, Raghu Murthy <[EMAIL PROTECTED]> wrote:

> Hive.java does not call methods in HiveServer. Can you explain what you are
> trying to achieve? You should just be able to start the HiveServer and then
> create a HiveClient to invoke methods of both HiveServer itself and
> HiveMetaStore.
>
> On 5/13/09 8:07 PM, "Min Zhou" <[EMAIL PROTECTED]> wrote:
>
> > Hey Raghu,
> >
> > Agreed with you. My question is ,  Hive.java code called by HiveServer,
> and
> > Hive.java would talked to HiveServer through a socket connection. Am I
> right?
> >
> > Min
> > On Thu, May 14, 2009 at 11:02 AM, Raghu Murthy <[EMAIL PROTECTED]>
> wrote:
> >> We have used thrift's inheritance feature where one service can extend
> >> another service, i.e., HiveServer extends HiveMetaStore. This means that
> >> 1. HiveServer exposes HiveMetaStore methods
> >> 2. HiveClient can be used to invoke methods exposed by both HiveServer
> and
> >> HiveMetaStore.
> >>
> >> Does that make sense?
> >>
> >>
> >> On 5/13/09 7:52 PM, "Min Zhou" <[EMAIL PROTECTED]> wrote:
> >>
> >>>> Sorry, I didn't precisely express my meaning.  I didn't talk about a
> >>>> supposition, but talked about a possible phenomenon may happen at
> hive.
> >>>> The hiveserver may both be a server and a client of itself,  you can
> look
> >>>> >> at
> >>>> the definition of HiveServer
> >>>>
> >>>> public static class HiveServerHandler extends HiveMetaStore.HMSHandler
> ...
> >>>> {
> >>>>    ...
> >>>> }
> >>>>
> >>>>
> >>>>
> >>>> It extends HiveMetaStore.HMSHandler, which provide the same function
> of
> >>>> metastore .
> >>>>
> >>>> public class HiveMetaStore extends ThriftHiveMetastore {
> >>>>
> >>>>     public static class HMSHandler extends FacebookBase implements
> >>>> ThriftHiveMetastore.Iface {
> >>>>      // omit
> >>>>     }
> >>>>
> >>>>     // main entry of the MetaStore but didn't be called at this case
> >>>>     public static void main(String[] args) {
> >>>>     ...
> >>>>     }
> >>>> }
> >>>>
> >>>>  there is no another standalone metastore server if you have up a
> >>> hiveserver,
> >>>> which is also a metaserver itself.
> >>>>
> >>>> Min
> >>>> On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <[EMAIL PROTECTED]>
> wrote:
> >>>>>>
> >>>>>>
> >>>>>> I don't think that would work, where would you store the meta-data
> for
> >>>>>> >>> the
> >>>>>> meta-data?
> >>>>>>
> >>>>>>
> >>>>>> Min Zhou wrote:
> >>>>>>>> Hi  Ashish,
> >>>>>>>> Thank you for your swift reply. I guess it's HiveServer code
> inherit
> >>>>>>>> the
> >>>>>>>> metastore thrift api. HiveServer provide metadata service by
> itself,
> >>>>>>>> not
> >>>>>>>> call another metastore server,  and Hive.class talks to the
> HiveServer
> >>>>>>>> to
> >>>>>>>> get
> >>>>>>>> the metadata.
> >>>>>>>> Regards,
> >>>>>>>> Min
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <
> [EMAIL PROTECTED]>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Apologies for the delay Min..
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> The HiveServer code calls the metastore thrift api to get the
> >>>>>> metadata. The
> >>>>>>>>>> Hive.class in ql also talks to the metastore api to get the
> metadata.
> >>>>>>>>>> Not
> >>>>>>>>>> sure if I follow your question though?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Ashish
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> From: Min Zhou [mailto:[EMAIL PROTECTED]]
> >>>>>>>>>>  Sent: Wednesday, May 13, 2009 6:07 PM
> >>>>>>>>>>  To: hive-user
> >>>>>>>>>>  Subject: Re: Is it possible hiveserver both be a server and a
> client
> >>>>>>>>>> of
> >>>>>>>>>> itself?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Is there anyone can answer my question?
> >>>>

My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

I see your point Min. Yes this is possible however only the metadata calls to the HiveServer should emanate from Hive.java. An execute should not call an execute etc... but it is possible in deployments where the HiveServer is its own clients and if I am correct thrift stubs should be able to deal with this and not make a network connection.

However, the ideal deployment would have a HiveServer in front of a metastore server (which is different). A cluster of HiveServers front a single metastore server. We would want the clustering on HiveServers as they serve out the data to clients talking jdbc. So in that deployment (which we think would make sense if  a deployment is large scale and hit by many users), the metadata calls made though Hive.java (in turn called though the execute call in HiveServer) would just go to a different metastore server.

Ashish

________________________________________
From: Min Zhou [[EMAIL PROTECTED]]
Sent: Wednesday, May 13, 2009 8:21 PM
To: [EMAIL PROTECTED]
Subject: Re: Is it possible hiveserver both be a server and a client of         itself?

HiveServer does call methods in Hive.java when compiling and planing a query command.  HiveServer talk to itself through Hive.java and a socket connection.
This is the logic of the current hive trunk code.

On Thu, May 14, 2009 at 11:16 AM, Raghu Murthy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hive.java does not call methods in HiveServer. Can you explain what you are
trying to achieve? You should just be able to start the HiveServer and then
create a HiveClient to invoke methods of both HiveServer itself and
HiveMetaStore.

On 5/13/09 8:07 PM, "Min Zhou" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:

> Hey Raghu,
>
> Agreed with you. My question is ,  Hive.java code called by HiveServer,  and
> Hive.java would talked to HiveServer through a socket connection. Am I right?
>
> Min
> On Thu, May 14, 2009 at 11:02 AM, Raghu Murthy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> We have used thrift's inheritance feature where one service can extend
>> another service, i.e., HiveServer extends HiveMetaStore. This means that
>> 1. HiveServer exposes HiveMetaStore methods
>> 2. HiveClient can be used to invoke methods exposed by both HiveServer and
>> HiveMetaStore.
>>
>> Does that make sense?
>>
>>
>> On 5/13/09 7:52 PM, "Min Zhou" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>
>>>> Sorry, I didn't precisely express my meaning.  I didn't talk about a
>>>> supposition, but talked about a possible phenomenon may happen at hive.
>>>> The hiveserver may both be a server and a client of itself,  you can look
>>>> >> at
>>>> the definition of HiveServer
>>>>
>>>> public static class HiveServerHandler extends HiveMetaStore.HMSHandler ...
>>>> {
>>>>    ...
>>>> }
>>>>
>>>>
>>>>
>>>> It extends HiveMetaStore.HMSHandler, which provide the same function of
>>>> metastore .
>>>>
>>>> public class HiveMetaStore extends ThriftHiveMetastore {
>>>>
>>>>     public static class HMSHandler extends FacebookBase implements
>>>> ThriftHiveMetastore.Iface {
>>>>      // omit
>>>>     }
>>>>
>>>>     // main entry of the MetaStore but didn't be called at this case
>>>>     public static void main(String[] args) {
>>>>     ...
>>>>     }
>>>> }
>>>>
>>>>  there is no another standalone metastore server if you have up a
>>> hiveserver,
>>>> which is also a metaserver itself.
>>>>
>>>> Min
>>>> On Thu, May 14, 2009 at 10:35 AM, Amr Awadallah <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>>>>>>
>>>>>>
>>>>>> I don't think that would work, where would you store the meta-data for
>>>>>> >>> the
>>>>>> meta-data?
>>>>>>
>>>>>>
>>>>>> Min Zhou wrote:
>>>>>>>> Hi  Ashish,
>>>>>>>> Thank you for your swift reply. I guess it's HiveServer code inherit
>>>>>>>> the
>>>>>>>> metastore thrift api. HiveServer provide metadata service by itself,
>>>>>>>> not
>>>>>>>> call another metastore server,  and Hive.class talks to the HiveServer
My research interests are distributed systems, parallel computing and bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

HiveServer is also a HiveMetastoreServer in usual configuration. But I think it is possible make HiveServer talk to a different server for metadata by changing hive-site.xml that HiveServer loads. I am not sure why it would be useful though. Though I didn't try this configuration.

Prasad

________________________________
From: Amr Awadallah <[EMAIL PROTECTED]>
Reply-To: <[EMAIL PROTECTED]>
Date: Wed, 13 May 2009 19:35:40 -0700
To: <[EMAIL PROTECTED]>
Subject: Re: Is it possible hiveserver both be a server and a client of itself?
I don't think that would work, where would you store the meta-data for the meta-data?

Min Zhou wrote:
Hi  Ashish,
Thank you for your swift reply. I guess it's HiveServer code inherit the metastore thrift api. HiveServer provide metadata service by itself,  not call another metastore server,  and Hive.class talks to the HiveServer to get the metadata.
Regards,
Min
On Thu, May 14, 2009 at 9:19 AM, Ashish Thusoo <[EMAIL PROTECTED]> wrote:

Apologies for the delay Min..

The HiveServer code calls the metastore thrift api to get the metadata. The Hive.class in ql also talks to the metastore api to get the metadata. Not sure if I follow your question though?

Ashish

________________________________
From: Min Zhou [mailto:[EMAIL PROTECTED]]
 Sent: Wednesday, May 13, 2009 6:07 PM
 To: hive-user
 Subject: Re: Is it possible hiveserver both be a server and a client of itself?
Is there anyone can answer my question?
Thanks in advanced!
On Sun, May 10, 2009 at 12:00 AM, Min Zhou <[EMAIL PROTECTED]> wrote:

Hi guys,

I found that HiveServer can fetch metadata thourgh Hive class after
generated a task like DDLTask, and meanwhile Hive class is a client
who ask HiveServer for metadata. So I guess it may be possible
hiveserver both be a server and a client of itself, am I right?
Thanks,
Min
 --
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
 http://www.linkedin.com/in/coderplay
My blog:
 http://coderplay.javaeye.com

--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.

My profile:
 http://www.linkedin.com/in/coderplay
My blog:
 http://coderplay.javaeye.com
--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.

My profile:
 http://www.linkedin.com/in/coderplay
My blog:
 http://coderplay.javaeye.com