Tianying Chang 2013-01-25, 18:54
Ted Yu 2013-01-25, 18:58
Tianying Chang 2013-01-25, 19:12
Marcos Ortiz 2013-01-25, 21:29
Tianying Chang 2013-01-25, 23:31
Shrijeet Paliwal 2013-01-25, 19:01
Tianying Chang 2013-01-26, 01:28
tsuna 2013-01-26, 06:53
-Re: AsynchBase client holds stale dead region server for long time even after the META has already been update.
As Shrijeet mentioned, we (@Rocketfuel) were experiencing this bug
internally when doing cluster restarts. After some trial and error, I was
able to create a set of steps to reproduce this bug in a controlled fashion
on our test cluster. Further, using heap dumps and added debug messages,
this looks like the cause and fix:
https://github.com/OpenTSDB/asynchbase/pull/48. I have tested this
repeatedly on the test cluster and things are looking fine. Please have a
look and see if this makes sense and if the fix is a correct one.
On Friday, 25 January 2013 22:53:17 UTC-8, tsuna wrote:
> > Thanks for the information! We have seen this couple times recently.
> Last week, it was very long(like 40+ minutes before we restart). I will
> follow up on that discuss thread. Thanks a lot!!
> This is bug number 1, I haven't been able to track it down as I've
> never been able to reproduce it in a controller fashion :(
> I also spent hours manually walking references of heap dumps
> and checking state to see if anything was wrong but I haven't
> found anything, not even a clue.
> Benoit "tsuna" Sigoure
Marcos Ortiz 2013-01-29, 14:07