I'm looking at the client code here:
https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client

Is this the high level operations, and the actual sending of this data over
the network is done somewhere else?

For example, during a PUT, you may want it to write to n nodes, where is
the code that does that? And the actual network connection etc?

Hi,

If you're speaking about preparing the query it's in HTable and
HConnectionManager.
If you're on the pure network level, then, on trunk, it's now done
with a third party called protobuf.

See the code from HConnectionManager#createCallable to see how it's used.

Cheers,

N.

On Tue, May 29, 2012 at 4:15 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> I'm looking at the client code here:
> https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client
>
> Is this the high level operations, and the actual sending of this data over
> the network is done somewhere else?
>
> For example, during a PUT, you may want it to write to n nodes, where is
> the code that does that? And the actual network connection etc?

So how does thrift and avro fit into the picture?  (I believe I saw
references to that somewhere, are those alternate connection libs?)

I know protobuf is just generating types for various languages...

On Tue, May 29, 2012 at 10:26 AM, N Keywal <[EMAIL PROTECTED]> wrote:

> Hi,
>
> If you're speaking about preparing the query it's in HTable and
> HConnectionManager.
> If you're on the pure network level, then, on trunk, it's now done
> with a third party called protobuf.
>
> See the code from HConnectionManager#createCallable to see how it's used.
>
> Cheers,
>
> N.
>
> On Tue, May 29, 2012 at 4:15 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> > I'm looking at the client code here:
> >
> https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client
> >
> > Is this the high level operations, and the actual sending of this data
> over
> > the network is done somewhere else?
> >
> > For example, during a PUT, you may want it to write to n nodes, where is
> > the code that does that? And the actual network connection etc?
>

There are two levels:
- communication between hbase client and hbase cluster: this is the
code you have in hbase client package. As a end user you don't really
care, but you care if you want to learn hbase internals.
- communication between customer code and hbase as a whole if you
don't want to use the hbase client. Then several options are
available, thrift being one of them (I'm not sure of avro status).

What do you want to do exactly?

On Tue, May 29, 2012 at 4:33 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> So how does thrift and avro fit into the picture?  (I believe I saw
> references to that somewhere, are those alternate connection libs?)
>
> I know protobuf is just generating types for various languages...
>
> On Tue, May 29, 2012 at 10:26 AM, N Keywal <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> If you're speaking about preparing the query it's in HTable and
>> HConnectionManager.
>> If you're on the pure network level, then, on trunk, it's now done
>> with a third party called protobuf.
>>
>> See the code from HConnectionManager#createCallable to see how it's used.
>>
>> Cheers,
>>
>> N.
>>
>> On Tue, May 29, 2012 at 4:15 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
>> > I'm looking at the client code here:
>> >
>> https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client
>> >
>> > Is this the high level operations, and the actual sending of this data
>> over
>> > the network is done somewhere else?
>> >
>> > For example, during a PUT, you may want it to write to n nodes, where is
>> > the code that does that? And the actual network connection etc?
>>

I don't really want any, I just want to learn the internals :)

So why would someone not want to use the client, for data intensive tasks
like mapreduce etc. where they want direct access to the files?

On Tue, May 29, 2012 at 11:00 AM, N Keywal <[EMAIL PROTECTED]> wrote:

> There are two levels:
> - communication between hbase client and hbase cluster: this is the
> code you have in hbase client package. As a end user you don't really
> care, but you care if you want to learn hbase internals.
> - communication between customer code and hbase as a whole if you
> don't want to use the hbase client. Then several options are
> available, thrift being one of them (I'm not sure of avro status).
>
> What do you want to do exactly?
>
> On Tue, May 29, 2012 at 4:33 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> > So how does thrift and avro fit into the picture?  (I believe I saw
> > references to that somewhere, are those alternate connection libs?)
> >
> > I know protobuf is just generating types for various languages...
> >
> > On Tue, May 29, 2012 at 10:26 AM, N Keywal <[EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> If you're speaking about preparing the query it's in HTable and
> >> HConnectionManager.
> >> If you're on the pure network level, then, on trunk, it's now done
> >> with a third party called protobuf.
> >>
> >> See the code from HConnectionManager#createCallable to see how it's
> used.
> >>
> >> Cheers,
> >>
> >> N.
> >>
> >> On Tue, May 29, 2012 at 4:15 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> >> > I'm looking at the client code here:
> >> >
> >>
> https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client
> >> >
> >> > Is this the high level operations, and the actual sending of this data
> >> over
> >> > the network is done somewhere else?
> >> >
> >> > For example, during a PUT, you may want it to write to n nodes, where
> is
> >> > the code that does that? And the actual network connection etc?
> >>
>

So it's the right place for the internals :-).
The main use case for the thrift api is when you have non java client code.

On Tue, May 29, 2012 at 5:07 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> I don't really want any, I just want to learn the internals :)
>
> So why would someone not want to use the client, for data intensive tasks
> like mapreduce etc. where they want direct access to the files?
>
> On Tue, May 29, 2012 at 11:00 AM, N Keywal <[EMAIL PROTECTED]> wrote:
>
>> There are two levels:
>> - communication between hbase client and hbase cluster: this is the
>> code you have in hbase client package. As a end user you don't really
>> care, but you care if you want to learn hbase internals.
>> - communication between customer code and hbase as a whole if you
>> don't want to use the hbase client. Then several options are
>> available, thrift being one of them (I'm not sure of avro status).
>>
>> What do you want to do exactly?
>>
>> On Tue, May 29, 2012 at 4:33 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
>> > So how does thrift and avro fit into the picture?  (I believe I saw
>> > references to that somewhere, are those alternate connection libs?)
>> >
>> > I know protobuf is just generating types for various languages...
>> >
>> > On Tue, May 29, 2012 at 10:26 AM, N Keywal <[EMAIL PROTECTED]> wrote:
>> >
>> >> Hi,
>> >>
>> >> If you're speaking about preparing the query it's in HTable and
>> >> HConnectionManager.
>> >> If you're on the pure network level, then, on trunk, it's now done
>> >> with a third party called protobuf.
>> >>
>> >> See the code from HConnectionManager#createCallable to see how it's
>> used.
>> >>
>> >> Cheers,
>> >>
>> >> N.
>> >>
>> >> On Tue, May 29, 2012 at 4:15 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
>> >> > I'm looking at the client code here:
>> >> >
>> >>
>> https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client
>> >> >
>> >> > Is this the high level operations, and the actual sending of this data
>> >> over
>> >> > the network is done somewhere else?
>> >> >
>> >> > For example, during a PUT, you may want it to write to n nodes, where
>> is
>> >> > the code that does that? And the actual network connection etc?
>> >>
>>

The way thrift and avro fits in here is ...

Thrift Client (your code) -> (thrift on the wire) -> Thrift Server
(provided by HBase) -> (uses HTable) -> HBase Cluster.

Same with Avro.

So - use HTable if you want to interact with the cluster using a Java
API ... use the others if you want non-Java clients to access HBase.

On Tue, May 29, 2012 at 7:33 AM, S Ahmed <[EMAIL PROTECTED]> wrote:
> So how does thrift and avro fit into the picture?  (I believe I saw
> references to that somewhere, are those alternate connection libs?)
>
> I know protobuf is just generating types for various languages...
>
> On Tue, May 29, 2012 at 10:26 AM, N Keywal <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> If you're speaking about preparing the query it's in HTable and
>> HConnectionManager.
>> If you're on the pure network level, then, on trunk, it's now done
>> with a third party called protobuf.
>>
>> See the code from HConnectionManager#createCallable to see how it's used.
>>
>> Cheers,
>>
>> N.
>>
>> On Tue, May 29, 2012 at 4:15 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
>> > I'm looking at the client code here:
>> >
>> https://github.com/apache/hbase/tree/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client
>> >
>> > Is this the high level operations, and the actual sending of this data
>> over
>> > the network is done somewhere else?
>> >
>> > For example, during a PUT, you may want it to write to n nodes, where is
>> > the code that does that? And the actual network connection etc?
>>