Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> hbase insert performance test (from hbasemaster and regionservers)


Copy link to this message
-
Re: hbase insert performance test (from hbasemaster and regionservers)
Hi,

Seems we just had someone talk about this just the other day...

1) 8GB of memory isn't enough to run both M/R and HBase.
Ok, yes you can run it, however don't expect it to perform well.

2) You never want a user to run their own code from the cluster itself. Use an *edge* node.

There's more, but you get the idea.

On May 21, 2012, at 2:04 PM, Faruk Berksöz wrote:

> Dear All,
> we have 4 node in our cluster (1nn+dn,3 dn).
> Hadoop dist. is cdh3u3.
> Every node has 2 tb disk , 8 gb memory.
> We are trying some insert performance test on hbase.
> I have tried to insert 250.000 records from hbase master  (without thread),
> that takes  5-7 sec.
> But when I try ro insert from  any regionserver the same data (250.000)  ,
> it takes longer  21 sec.
> Is that normal ?
>
> any response would be appreciated..
>
>
> my java code looks like:
>            .........
>            .........
>            .........
>            long start = System.currentTimeMillis();
>            //DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd
> HH:mm:ss");
>            //Date sdate = new Date();
>
>            //LOG.info(" Start-Time :" + dateFormat.format(sdate));
>            for (int i = 0; i < eachsize; i++) {
>
>
>                put = new Put(String.format("%016x",
> random.nextLong()).getBytes());
>
>                put.setWriteToWAL(false);
>                //set column and their values
>                addColumnAndValues();
>                table.put(put);
>
>
>            }
>            elapsedTimeMillis = System.currentTimeMillis() - start;
>            elapsedTimeSec = elapsedTimeMillis / 1000F;
>            elapsedTimeSecInMemory = elapsedTimeSec ;
>            LOG.info(" Elapsed-Time (sec) inMemory :" + elapsedTimeSec);
>
>            table.flushCommits();
>            //disk write elapsed time
>            elapsedTimeMillis = System.currentTimeMillis() - start;
>            elapsedTimeSec = elapsedTimeMillis / 1000F;
>            elapsedTimeSecDiskWrite   = elapsedTimeSec;
>            LOG.info(" Elapsed-Time (sec) Disk Write:" +
> elapsedTimeSec);
>            .........
>            .........
>
> My hbase-site.xml is looks like :
> <configuration>
>    <property>
>        <name>hbase.rootdir</name>
>        <value>hdfs://master.bigdata.com:54310/hbase</value>
>    </property>
>    <property>
>        <name>hbase.cluster.distributed</name>
>        <value>true</value>
>    </property>
>    <property>
>        <name>hbase.zookeeper.quorum</name>
>        <value>master.bigdata.com,slave1.bigdata.com,slave2.bigdata.com,
> slave3.bigdata.com</value>
>    </property>
>    <property>
>        <name>hbase.zookeeper.dns.interface</name>
>        <value>eth0</value>
>    </property>
>    <property>
>        <name>hbase.zookeeper.dns.nameserver</name>
>        <value>10.10.10.1</value>
>    </property>
>
>    <property>
>        <name>hbase.regionserver.handler.count</name>
>        <value>20</value>
>    </property>
>
>    <property>
>        <name>hbase.client.write.buffer</name>
>        <value>5097152</value>
>    </property>
> </configuration>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB