Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Compaction problem


Copy link to this message
-
Re: Compaction problem
Thanks Asaf.

I alredy verified everythin with ethtool and netperf, it was realy the
hadoop speed I was looking for. I got 45.31 MB/sec when I tried to put
a 124GB file. I will try to find how/where this can be improved, but
most probably in another mailing list ;)

JM

2013/3/27 Asaf Mesika <[EMAIL PROTECTED]>:
> From my testing: I had 3 RS and 1 Master all connected to a single router at speed of 1000 mbps.
> This network measured speed was 107 - 120 MB/sec, while the theoretic max is 125 MB/sec (1000/8)
> HDFS Write speed (copyFromLocal) was 73 MB/sec
> HDFS Read speed (copyToLocal) was 97 MB/sec
>
> Now, if you're writing to HBase constantly, you basically get couple of HDFS clients happening in the same site:
> 1. WAL writing
> 2. Flush
> 3. Compaction
>
> So, if you can write to HDFS at 73 MB/sec, you will get ~ 25 MB/sec. I didn't get that - I guess HBase process it self add some to it, so I got around 18 MB/sec.
>
> All this testing was done using HBase vanilla. All computers had 4 cores each 3.1 GHz, and 8GB of RAM.
>
>
> I used ethtool to test if my computer was connected to the router at rate of 1000 Mbit/sec
> I used netperf to measure network speed between each data node (You may find one link which is faulty out of the 3)
>
> You can also use iozone to check if the hard-drives are the one causing the bottleneck.
> (I used the following command: iozone -R -r 16384k -s 64m -l 1 -u 1 -F ~/f1 -I)
> For instance, my hard-drives gave mixed read-write rate between 90-120 MB/sec.
>
> Thanks,
> Asaf
>
> On Mar 27, 2013, at 5:05 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]> wrote:
>
>> Hi Asaf,
>>
>> What kind of results should we expect from the test you are suggesting?
>>
>> I mean, how many MB/sec should we see on an healthy cluster?
>>
>> Thanks,
>>
>> JM
>>
>> 2013/3/26 Asaf Mesika <[EMAIL PROTECTED]>:
>>> 1st thing I would do to find the bottleneck it so benchmark HDFS solo performance.
>>> Create a 16GB file (using dd) which is x2 your memory and run "time hadoop fs -copyFromLocal yourFile.txt /tmp/a.txt"
>>> Tell us what is the speed of this file copy in MB/sec.
>>>
>>>
>>> On Mar 22, 2013, at 4:44 PM, tarang dawer <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi
>>>> As per my use case , I have to write around 100gb data , with a ingestion
>>>> speed of around 200 mbps. While writing , i am getting a performance hit by
>>>> compaction , which adds to the delay.
>>>> I am using a 8 core machine with 16 gb RAM available., 2 Tb hdd 7200RPM.
>>>> Got some idea from the archives and  tried pre splitting the regions ,
>>>> configured HBase with following parameters(configured the parameters in a
>>>> haste , so please guide me if anything's out of order) :-
>>>>
>>>>
>>>>       <property>
>>>>               <name>hbase.hregion.memstore.block.multiplier</name>
>>>>               <value>4</value>
>>>>       </property>
>>>>       <property>
>>>>                <name>hbase.hregion.memstore.flush.size</name>
>>>>                <value>1073741824</value>
>>>>       </property>
>>>>
>>>>       <property>
>>>>               <name>hbase.hregion.max.filesize</name>
>>>>               <value>1073741824</value>
>>>>       </property>
>>>>       <property>
>>>>               <name>hbase.hstore.compactionThreshold</name>
>>>>               <value>5</value>
>>>>       </property>
>>>>       <property>
>>>>             <name>hbase.hregion.majorcompaction</name>
>>>>                 <value>0</value>
>>>>       </property>
>>>>       <property>
>>>>               <name>hbase.hstore.blockingWaitTime</name>
>>>>               <value>30000</value>
>>>>       </property>
>>>>        <property>
>>>>                <name>hbase.hstore.blockingStoreFiles</name>
>>>>                <value>200</value>
>>>>        </property>
>>>>
>>>> <property>
>>>>       <name>hbase.regionserver.lease.period</name>
>>>>       <value>3000000</value>
>>>> </property>
>>>>
>>>>
>>>> but still m not able to achieve the optimal rate , getting around 110 mbps.