Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - how many severs in a hbase cluster


+
myhbase 2013-06-22, 14:21
+
Jean-Marc Spaggiari 2013-06-22, 14:29
+
Mohammad Tariq 2013-06-22, 14:35
+
Mohammad Tariq 2013-06-22, 14:37
+
myhbase 2013-06-22, 15:01
+
Mohammad Tariq 2013-06-22, 15:56
+
Jean-Marc Spaggiari 2013-06-22, 16:09
+
Kevin Odell 2013-06-22, 16:15
+
Mohammad Tariq 2013-06-22, 17:05
+
iain wright 2013-06-22, 22:37
+
Mohammad Tariq 2013-06-22, 23:21
Copy link to this message
-
Re: how many severs in a hbase cluster
Kevin O'dell 2013-06-23, 13:39
Mohammad,

  The NN is low write, and has pretty static memory usage.  You will see
the NN memory usage go up as you add blocks/files.  Since, HBase has memory
limitations(GC's Fault), and should have ~1 file per store you will not
have a lot of memory pressure on the NN.  The JT is the same way, it scales
up usage based on number of MR jobs.  In a sane HBase environment you are
not going to be running 1000s of MR jobs against HBase.  ZK also has pretty
minimum requirements - 1GB of memory, dedicated CPU core, and place to
write to with low I/O wait.  I have always found the NN, SNN, and JT to be
the next best place to put the ZK if dedicated HW is not available.  I have
seen some strange behavior with ZK runs on DN/TT/RS nodes.  From
unexplained timeouts to corrupt znodes causing failures(This one was real
nasty).
On Sat, Jun 22, 2013 at 7:21 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Hello Iain,
>
>          You would put a lot of pressure on the RAM if you do that. NN
> already has high memory requirement and then having JT+ZK on the same
> machine would be too heavy, IMHO.
>
> Warm Regards,
> Tariq
> cloudfront.blogspot.com
>
>
> On Sun, Jun 23, 2013 at 4:07 AM, iain wright <[EMAIL PROTECTED]> wrote:
>
> > Hi Mohammad,
> >
> > I am curious why you chose not to put the third ZK on the NN+JT? I was
> > planning on doing that on a new cluster and want to confirm it would be
> > okay.
> >
> >
> > --
> > Iain Wright
> > Cell: (562) 852-5916
> >
> > <http://www.labctsi.org/>
> > This email message is confidential, intended only for the recipient(s)
> > named above and may contain information that is privileged, exempt from
> > disclosure under applicable law. If you are not the intended recipient,
> do
> > not disclose or disseminate the message to anyone except the intended
> > recipient. If you have received this message in error, or are not the
> named
> > recipient(s), please immediately notify the sender by return email, and
> > delete all copies of this message.
> >
> >
> > On Sat, Jun 22, 2013 at 10:05 AM, Mohammad Tariq <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Yeah, I forgot to mention that no. of ZKs should be odd. Perhaps those
> > > parentheses made that statement look like an optional statement. Just
> to
> > > clarify it was mandatory.
> > >
> > > Warm Regards,
> > > Tariq
> > > cloudfront.blogspot.com
> > >
> > >
> > > On Sat, Jun 22, 2013 at 9:45 PM, Kevin O'dell <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > If you run ZK with a DN/TT/RS please make sure to dedicate a hard
> drive
> > > and
> > > > a core to the ZK process. I have seen many strange occurrences.
> > > > On Jun 22, 2013 12:10 PM, "Jean-Marc Spaggiari" <
> > [EMAIL PROTECTED]
> > > >
> > > > wrote:
> > > >
> > > > > You HAVE TO run a ZK3, or else you don't need to have ZK2 and any
> ZK
> > > > > failure will be an issue. You need to have an odd number of ZK
> > > > > servers...
> > > > >
> > > > > Also, if you don't run MR jobs, you don't need the TT and JT...
> Else,
> > > > > everything below is correct. But there is many other options, all
> > > > > depend on your needs and the hardware you have ;)
> > > > >
> > > > > JM
> > > > >
> > > > > 2013/6/22 Mohammad Tariq <[EMAIL PROTECTED]>:
> > > > > > With 8 machines you can do something like this :
> > > > > >
> > > > > > Machine 1 - NN+JT
> > > > > > Machine 2 - SNN+ZK1
> > > > > > Machine 3 - HM+ZK2
> > > > > > Machine 4-8 - DN+TT+RS
> > > > > > (You can run ZK3 on a slave node with some additional memory).
> > > > > >
> > > > > > DN and RS run on the same machine. Although RSs are said to hold
> > the
> > > > > data,
> > > > > > the data is actually stored in DNs. Replication is managed at
> HDFS
> > > > level.
> > > > > > You don't have to worry about that.
> > > > > >
> > > > > > You can visit this link <
> > > > http://hbase.apache.org/book/perf.writing.html>
> > > > > to
> > > > > > see how to write efficiently into HBase. With a small field there
> > > > should
>
Kevin O'Dell
Systems Engineer, Cloudera