Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> how many severs in a hbase cluster


+
myhbase 2013-06-22, 14:21
+
Jean-Marc Spaggiari 2013-06-22, 14:29
+
Mohammad Tariq 2013-06-22, 14:35
+
Mohammad Tariq 2013-06-22, 14:37
+
myhbase 2013-06-22, 15:01
+
Mohammad Tariq 2013-06-22, 15:56
+
Jean-Marc Spaggiari 2013-06-22, 16:09
+
Kevin Odell 2013-06-22, 16:15
+
Mohammad Tariq 2013-06-22, 17:05
+
iain wright 2013-06-22, 22:37
+
Mohammad Tariq 2013-06-22, 23:21
Copy link to this message
-
Re: how many severs in a hbase cluster
Mohammad,

  The NN is low write, and has pretty static memory usage.  You will see
the NN memory usage go up as you add blocks/files.  Since, HBase has memory
limitations(GC's Fault), and should have ~1 file per store you will not
have a lot of memory pressure on the NN.  The JT is the same way, it scales
up usage based on number of MR jobs.  In a sane HBase environment you are
not going to be running 1000s of MR jobs against HBase.  ZK also has pretty
minimum requirements - 1GB of memory, dedicated CPU core, and place to
write to with low I/O wait.  I have always found the NN, SNN, and JT to be
the next best place to put the ZK if dedicated HW is not available.  I have
seen some strange behavior with ZK runs on DN/TT/RS nodes.  From
unexplained timeouts to corrupt znodes causing failures(This one was real
nasty).
On Sat, Jun 22, 2013 at 7:21 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Hello Iain,
>
>          You would put a lot of pressure on the RAM if you do that. NN
> already has high memory requirement and then having JT+ZK on the same
> machine would be too heavy, IMHO.
>
> Warm Regards,
> Tariq
> cloudfront.blogspot.com
>
>
> On Sun, Jun 23, 2013 at 4:07 AM, iain wright <[EMAIL PROTECTED]> wrote:
>
> > Hi Mohammad,
> >
> > I am curious why you chose not to put the third ZK on the NN+JT? I was
> > planning on doing that on a new cluster and want to confirm it would be
> > okay.
> >
> >
> > --
> > Iain Wright
> > Cell: (562) 852-5916
> >
> > <http://www.labctsi.org/>
> > This email message is confidential, intended only for the recipient(s)
> > named above and may contain information that is privileged, exempt from
> > disclosure under applicable law. If you are not the intended recipient,
> do
> > not disclose or disseminate the message to anyone except the intended
> > recipient. If you have received this message in error, or are not the
> named
> > recipient(s), please immediately notify the sender by return email, and
> > delete all copies of this message.
> >
> >
> > On Sat, Jun 22, 2013 at 10:05 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Yeah, I forgot to mention that no. of ZKs should be odd. Perhaps those
> > > parentheses made that statement look like an optional statement. Just
> to
> > > clarify it was mandatory.
> > >
> > > Warm Regards,
> > > Tariq
> > > cloudfront.blogspot.com
> > >
> > >
> > > On Sat, Jun 22, 2013 at 9:45 PM, Kevin O'dell <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > If you run ZK with a DN/TT/RS please make sure to dedicate a hard
> drive
> > > and
> > > > a core to the ZK process. I have seen many strange occurrences.
> > > > On Jun 22, 2013 12:10 PM, "Jean-Marc Spaggiari" <
> > [EMAIL PROTECTED]
> > > >
> > > > wrote:
> > > >
> > > > > You HAVE TO run a ZK3, or else you don't need to have ZK2 and any
> ZK
> > > > > failure will be an issue. You need to have an odd number of ZK
> > > > > servers...
> > > > >
> > > > > Also, if you don't run MR jobs, you don't need the TT and JT...
> Else,
> > > > > everything below is correct. But there is many other options, all
> > > > > depend on your needs and the hardware you have ;)
> > > > >
> > > > > JM
> > > > >
> > > > > 2013/6/22 Mohammad Tariq <[EMAIL PROTECTED]>:
> > > > > > With 8 machines you can do something like this :
> > > > > >
> > > > > > Machine 1 - NN+JT
> > > > > > Machine 2 - SNN+ZK1
> > > > > > Machine 3 - HM+ZK2
> > > > > > Machine 4-8 - DN+TT+RS
> > > > > > (You can run ZK3 on a slave node with some additional memory).
> > > > > >
> > > > > > DN and RS run on the same machine. Although RSs are said to hold
> > the
> > > > > data,
> > > > > > the data is actually stored in DNs. Replication is managed at
> HDFS
> > > > level.
> > > > > > You don't have to worry about that.
> > > > > >
> > > > > > You can visit this link <
> > > > http://hbase.apache.org/book/perf.writing.html>
> > > > > to
> > > > > > see how to write efficiently into HBase. With a small field there
> > > > should
>
Kevin O'Dell
Systems Engineer, Cloudera
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB