Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Accumulo >> mail # user >> Uneven distribute of Hosted Tablets?


Copy link to this message
-
Re: Uneven distribute of Hosted Tablets?
So (at the risk of stating the obvious) it seems like your cluster is in a
funny state.  I would expect the counts in the "Hosted Tablets" column to
all be roughly the same, especially after restarting the master, assuming
you're using one of the balancers that comes with Accumulo.  It's possible
the cluster has gotten into this state due to the clock differences.
Accumulo has a mechanism called "logical time" to deal with clock
differences, but it is not enabled by default.  You can enable it when you
create a table.  If you don't enable this it is recommended that you use
NTP to synchronize the clocks on your cluster.  The !METADATA table has
logical time by default, but your other tables might not contain what you
expect them to if you haven't enabled logical time.

That said, I'm not sure why the clock issue would be affecting the
balancing.  You mentioned the new warnings you saw on the monitor page
after you restarted the system.  Could you see if there are any older
errors in your log files?

Billie
On Fri, May 31, 2013 at 8:10 AM, Ott, Charles H. <[EMAIL PROTECTED]>wrote:

> -bash-4.1$ ssh 1620-accumulo****
>
> -bash-4.1$ date****
>
> Fri May 31 *10:52:49 *EDT 2013****
>
> ** **
>
> -bash-4.1$ ssh 1620-Node1****
>
> -bash-4.1$ date****
>
> Fri May 31 *11:05:48* EDT 2013****
>
> ** **
>
> -bash-4.1$ ssh 1620-Node2****
>
> -bash-4.1$ date****
>
> Fri May 31 *11:05:58* EDT 2013****
>
> ** **
>
> -bash-4.1$ ssh 1620-Node3****
>
> -bash-4.1$ date****
>
> Fri May 31 *11:05:58* EDT 2013****
>
> ** **
>
> Looks like the master(1620-accumulo) and it’s tablet server are 12-13
> minutes behind the nodes.  I’m not sure my
> zookeeper+Hadoop+Accumulo+storm+Kafka stack will appreciate moving forward
> in time 12 minutes.  ****
>
> ** **
>
> *From:* [EMAIL PROTECTED][mailto:
> [EMAIL PROTECTED]] *On Behalf
> Of *Billie Rinaldi
> *Sent:* Friday, May 31, 2013 11:02 AM
> *To:* [EMAIL PROTECTED]
>
> *Subject:* Re: Uneven distribute of Hosted Tablets?****
>
> ** **
>
> Those last contact times are concerning as well.  Have they always looked
> like that?  I notice they were roughly the same on your first screenshot.
> Are your server clocks not in sync?****
>
> Billie****
>
> ** **
>
> On Fri, May 31, 2013 at 7:00 AM, Ott, Charles H. <[EMAIL PROTECTED]>
> wrote:****
>
> I performed a clean shutdown and startup of all the processes using the
> start-all.sh/stop-all.sh scripts.****
>
>  ****
>
> The systems have only been online for about 5 minutes and everything is
> working.  But I see the following Recent WARN in the Logs:****
>
>  ****
>
> time
> application                          count    level      message****
>
> 31 09:37:57,0774               tserver:1620-accumulo  1
> WARN   Future location is not to this server for the root tablet****
>
>  ****
>
> Hosted tablet distribution seems to be worse:****
>
>  ****
>
> (Image Below Here)****
>
>
> (Image Above Here)****
>
>  ****
>
> I am able to login and scans seems to be responsive.   I noticed that when
> we had our entries ~20 M count, our batch scans were taking much longer.  I
> was hoping that by distributing the tablets evenly, and splitting some of
> the bigger tables, we could get better performance.****
>
> As for splitting the bigger table, I received a message from a peer.  He
> mentioned that I could create a new table and split it on the values I
> want.  Then use Map reduce job to move the data from the single tablet
> table to split table.  ****
>
>  ****
>
> *From:* [EMAIL PROTECTED][mailto:
> [EMAIL PROTECTED]] *On Behalf
> Of *John Vines
> *Sent:* Thursday, May 30, 2013 5:30 PM
> *To:* [EMAIL PROTECTED]
> *Cc:* Lahr-Vivaz, Emilio F.****
>
>
> *Subject:* Re: Uneven distribute of Hosted Tablets?****
>
>  ****
>
> Your distribution is cause for concern. I thought we had resolved a lot of
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB