Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Will hbase automatically distribute the data across region servers or NOT..??


+
Vamshi Krishna 2013-08-23, 11:47
+
Samir Ahmic 2013-08-23, 12:26
+
Stack 2013-08-23, 15:21
+
Frank Chow 2013-08-23, 13:30
+
Vamshi Krishna 2013-08-23, 13:38
+
Dhaval Shah 2013-08-23, 14:10
+
Vamshi Krishna 2013-08-23, 14:21
+
Mike Axiak 2013-08-23, 14:23
Copy link to this message
-
Re: Will hbase automatically distribute the data across region servers or NOT..??
Dhaval Shah 2013-08-23, 14:49
Ok. The balancer runs as a separate thread (there is a config to set how often the thread wakes up but can't remember off the top of my head). Maybe if you wait long enough, it will balance eventually. Another thing you can try is run the balancer from hbase shell and see what you get back. If you get back a true, it means it should balance. If you get back a false, look at hbase master logs to see whats happening. I once had a scenario where my Unix accounts were messed up (2 users - hbase and another user mapped to the same unix ID and HDFS thought the user did not have the permissions to write to the HBase files on HDFS) and balancer did not run due to this exception. 

Another thing is (I think!) balancer generally does not run when regions are splitting. So its possible in your case that your regions are splitting so often (due to 10MB limit) that the balancer cannot be run since your regions are not stationary
Regards,
Dhaval
________________________________
From: Vamshi Krishna <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Dhaval Shah <[EMAIL PROTECTED]>
Sent: Friday, 23 August 2013 10:21 AM
Subject: Re: Will hbase automatically distribute the data across region servers or NOT..??
No that is 10MB itself. Just to observe the region splitting with respect
to the amount of data i am inserting in to hbase.
So, here i am inserting 40-50mb data and fixing that property to 10mb and
checking the region splitting happening.
But the intersting thing is regions got split BUT they are not being
distributed across other servers.
Whatever regions formed from the created tables on machine-1, all of them
are residing on the same machine-1 not being moved to other machine.
On Fri, Aug 23, 2013 at 7:40 PM, Dhaval Shah <[EMAIL PROTECTED]>wrote:

> Vamshi, max value for hbase.hregion.max.filesize to 10MB seems too small.
> Did you mean 10GB?
>
>
> Regards,
> Dhaval
>
>
> ________________________________
> From: Vamshi Krishna <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; zhoushuaifeng <[EMAIL PROTECTED]>
> Sent: Friday, 23 August 2013 9:38 AM
> Subject: Re: Will hbase automatically distribute the data across region
> servers or NOT..??
>
>
> Thanks for the clarifications.
> I am using hbase-0.94.10 and zookeepr-3.4.5
> But I am running into different issues .
> I set  hbase.hregion.max.filesize to 10Mb and i am inserting 10 million
> rows in to hbase table. During the insertion after some time, suddenly
> master is going down. I don't know what is the reason for such peculiar
> behavior.
> I found in master log below content and not able to make out what exactly
> the mistake. Please somebody help.
>
> master-log:
>
> 2013-08-23 18:56:36,865 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> 2013-08-23 18:56:36,866 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state :
>
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
> Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
>
> scores,\x00\x00\x00\x00\x00\x02\xC8t,1377264003140.a564f31795091b6513880c5db49ec90f.
> state=PENDING_OPEN, ts=1377264396861, server=vamshi,60020,1377263789273 ..
> Cannot transit it to OFFLINE.
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
>     at
>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
>     at
>
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
>     at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)

*Regards*
*
Vamshi Krishna

+
Frank Chow 2013-08-25, 11:23
+
Vamshi Krishna 2013-08-26, 06:23
+
Frank Chow 2013-08-28, 13:34