Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - HBase Is So Slow To Save Data?


Copy link to this message
-
Re: HBase Is So Slow To Save Data?
Bing Li 2012-08-29, 14:08
Dear all,

By the way, my HBase is in the pseudo-distributed mode. Thanks!

Best regards,
Bing

On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:

> Dear all,
>
> According to my experiences, it is very slow for HBase to save data? Am I
> right?
>
> For example, today I need to save data in a HashMap to HBase. It took
> about more than three hours. However when saving the same HashMap in a file
> in the text format with the redirected System.out, it took only 4.5 seconds!
>
> Why is HBase so slow? It is indexing?
>
> My code to save data in HBase is as follows. I think the code must be
> correct.
>
>         ......
>         public synchronized void
> AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
> ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int timingScale)
>         {
>                 List<Put> puts = new ArrayList<Put>();
>
>                 String hhNeighborRowKey;
>                 Put hubKeyPut;
>                 Put groupKeyPut;
>                 Put topGroupKeyPut;
>                 Put timingScalePut;
>                 Put nodeKeyPut;
>                 Put hubNeighborTypePut;
>
>                 for (Map.Entry<String, ConcurrentHashMap<String,
> Set<String>>> sourceHubGroupNeighborEntry : hhOutNeighborMap.entrySet())
>                 {
>                         for (Map.Entry<String, Set<String>>
> groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
>                         {
>                                 for (String neighborKey :
> groupNeighborEntry.getValue())
>                                 {
>                                         hhNeighborRowKey > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
> Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
> groupNeighborEntry.getKey() + timingScale + neighborKey);
>
>                                         hubKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
> Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
>                                         puts.add(hubKeyPut);
>
>                                         groupKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
> Bytes.toBytes(groupNeighborEntry.getKey()));
>                                         puts.add(groupKeyPut);
>
>                                         topGroupKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> topGroupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TOP_GROUP_KEY_COLUMN),
> Bytes.toBytes(GroupRegistry.WWW().GetParentGroupKey(groupNeighborEntry.getKey())));
>                                         puts.add(topGroupKeyPut);
>
>                                         timingScalePut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> timingScalePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TIMING_SCALE_COLUMN),
> Bytes.toBytes(timingScale));
>                                         puts.add(timingScalePut);
>
>                                         nodeKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> nodeKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_NODE_KEY_COLUMN),
> Bytes.toBytes(neighborKey));
>                                         puts.add(nodeKeyPut);
>
>                                         hubNeighborTypePut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> hubNeighborTypePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TYPE_COLUMN),
> Bytes.toBytes(SocialRole.VIRTUAL_NEIGHBOR));
>                                         puts.add(hubNeighborTypePut);