Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> AsynchBase client holds stale dead region server for long time even after the META has already been update.


+
Tianying Chang 2013-01-25, 18:54
+
Ted Yu 2013-01-25, 18:58
+
Tianying Chang 2013-01-25, 19:12
+
Marcos Ortiz 2013-01-25, 21:29
+
Tianying Chang 2013-01-25, 23:31
+
Shrijeet Paliwal 2013-01-25, 19:01
+
Tianying Chang 2013-01-26, 01:28
+
tsuna 2013-01-26, 06:53
Copy link to this message
-
Re: AsynchBase client holds stale dead region server for long time even after the META has already been update.
Hi Tsuna,
As Shrijeet mentioned, we (@Rocketfuel) were experiencing this bug
internally when doing cluster restarts. After some trial and error, I was
able to create a set of steps to reproduce this bug in a controlled fashion
on our test cluster. Further, using heap dumps and added debug messages,
this looks like the cause and fix:
https://github.com/OpenTSDB/asynchbase/pull/48. I have tested this
repeatedly on the test cluster and things are looking fine. Please have a
look and see if this makes sense and if the fix is a correct one.

Cheers,
Ishan

On Friday, 25 January 2013 22:53:17 UTC-8, tsuna wrote:
>
> On Fri, Jan 25, 2013 at 5:28 PM, Tianying Chang <[EMAIL PROTECTED]<javascript:>>
> wrote:
> > Thanks for the information! We have seen this couple times recently.
> Last week, it was very long(like 40+ minutes before we restart). I will
> follow up on that discuss thread. Thanks a lot!!
>
> This is bug number 1, I haven't been able to track it down as I've
> never been able to reproduce it in a controller fashion :(
> https://github.com/OpenTSDB/asynchbase/issues/1
>
> I also spent hours manually walking references of heap dumps
> and checking state to see if anything was wrong but I haven't
> found anything, not even a clue.
>
> --
> Benoit "tsuna" Sigoure
>
+
Marcos Ortiz 2013-01-29, 14:07
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB