Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> HConnectionManager$HConnectionImplementation.locateRegionInMeta


+
Kireet 2013-05-30, 03:51
+
Anoop John 2013-05-30, 04:16
+
Kireet 2013-05-30, 13:12
+
Ted Yu 2013-05-30, 13:20
+
Kireet 2013-05-30, 14:01
+
Ted Yu 2013-05-30, 14:26
+
Himanshu Vashishtha 2013-05-30, 16:48
+
Kireet 2013-05-30, 18:28
+
Ted Yu 2013-05-30, 23:38
+
Kireet 2013-05-31, 18:58
Copy link to this message
-
Re: HConnectionManager$HConnectionImplementation.locateRegionInMeta
Indeed. That is bad.
I cannot see a clean fix immediately, but we need to look at this.

Mind filing a ticket, Kireet?

-- Lars

________________________________
 From: Kireet <[EMAIL PROTECTED]>
To: public-user-50Pas4EWwPEyzMRdD/[EMAIL PROTECTED]
Sent: Friday, May 31, 2013 11:58 AM
Subject: Re: HConnectionManager$HConnectionImplementation.locateRegionInMeta
 

Even if I initiate the call via a pooled htable, the MetaScanner seems
to use a concrete HTable instance. The constructor invoked seems to
create a java ThreadPoolExecutor. I am not 100% sure but I think as long
as nothing is submitted to the ThreadPoolExecutor it won't create any
threads. I just wanted to confirm this was the case. I do see the
connection is shared.

--Kireet

On 5/30/13 7:38 PM, Ted Yu wrote:
> HTablePool$**PooledHTable is a wrapper around HTable.
>
> Here is how HTable obtains a connection:
>
>    public HTable(Configuration conf, final byte[] tableName, final
> ExecutorService pool)
>        throws IOException {
>      this.connection = HConnectionManager.getConnection(conf);
>
> Meaning the connection is a shared one based on certain key/value pairs
> from conf.
>
> bq. So every call to batch will create a new thread?
>
> I don't think so.
>
> On Thu, May 30, 2013 at 11:28 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+[EMAIL PROTECTED]> wrote:
>
>>
>>
>> Thanks, will give it a shot. So I should download 0.94.7 (latest stable)
>> and run the patch tool on top with the backport? This is a little new to me.
>>
>> Also, I was looking at the stack below. From my reading of the code, the
>> HTable.batch() call will always cause the prefetch call to occur, which
>> will cause a new HTable object to get created. The constructor used in
>> creating a new thread pool. So every call to batch will create a new
>> thread? Or the HTable's thread pool never gets used as the pool is only
>> used for writes? I think I am missing something but just want to confirm.
>>
>> Thanks
>> Kireet
>>
>> On 5/30/13 12:48 PM, Himanshu Vashishtha wrote:
>>
>>> bq. Anoop attached backported patch in HBASE-8655. It should go into
>>>
>>> 0.94.9, the next release - current is 0.94.8
>>>
>>> In case you want it sooner, you can apply 8655 patch and test/verify it.
>>>
>>> Thanks,
>>> Himanshu
>>>
>>>
>>>
>>> On Thu, May 30, 2013 at 7:26 AM, Ted Yu <yuzhihong-**
>>> Re5JQEeQqe8AvxtiuMwx3w@public.**gmane.org<[EMAIL PROTECTED]>>
>>> wrote:
>>>
>>>   Anoop attached backported patch in HBASE-8655
>>>>
>>>> It should go into 0.94.9, the next release - current is 0.94.8
>>>>
>>>> Cheers
>>>>
>>>> On Thu, May 30, 2013 at 7:01 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
>>>> [EMAIL PROTECTED] <kireet-Teh5dPVPL8nQT0dZR%[EMAIL PROTECTED]>>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> How long do backports typically take? We have to go live in a month
>>>>> ready
>>>>> or not. Thanks for the quick replies Anoop and Ted.
>>>>>
>>>>> --Kireet
>>>>>
>>>>>
>>>>> On 5/30/13 9:20 AM, Ted Yu wrote:
>>>>>
>>>>>   0.95 client is not compatible with 0.94 cluster. So you cannot use 0.95
>>>>>> client.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On May 30, 2013, at 6:12 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
>>>>>> AlfA-XMD5yJDbdMReXY1tMh2IBg@**public.gmane.org<[EMAIL PROTECTED]><
>>>>>> kireet-Teh5dPVPL8nQT0dZR%**2BAlfA-XMD5yJDbdMReXY1tMh2IBg@**
>>>>>> public.gmane.org<kireet-Teh5dPVPL8nQT0dZR%[EMAIL PROTECTED]>
>>>>>>
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Would there be a problem if our cluster is 0.94 and we use a 0.95
>>>>>>>
>>>>>> client?
>>>>
>>>>>
>>>>>>> I am not familiar with the HBase code base, but I did a dump of the
>>>>>>> thread that is actually running (below). It seems like it is related
>>>>>>>
>>>>>> to the
>>>>
>>>>> issue you mentioned as the running thread is doing the prefetch logic.
+
Kireet 2013-06-06, 15:25
+
Ted Yu 2013-05-30, 04:23
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB