Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase Is So Slow To Save Data?


Copy link to this message
-
HBase Is So Slow To Save Data?
Dear all,

According to my experiences, it is very slow for HBase to save data? Am I
right?

For example, today I need to save data in a HashMap to HBase. It took about
more than three hours. However when saving the same HashMap in a file in
the text format with the redirected System.out, it took only 4.5 seconds!

Why is HBase so slow? It is indexing?

My code to save data in HBase is as follows. I think the code must be
correct.

        ......
        public synchronized void
AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int timingScale)
        {
                List<Put> puts = new ArrayList<Put>();

                String hhNeighborRowKey;
                Put hubKeyPut;
                Put groupKeyPut;
                Put topGroupKeyPut;
                Put timingScalePut;
                Put nodeKeyPut;
                Put hubNeighborTypePut;

                for (Map.Entry<String, ConcurrentHashMap<String,
Set<String>>> sourceHubGroupNeighborEntry : hhOutNeighborMap.entrySet())
                {
                        for (Map.Entry<String, Set<String>>
groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
                        {
                                for (String neighborKey :
groupNeighborEntry.getValue())
                                {
                                        hhNeighborRowKey NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
groupNeighborEntry.getKey() + timingScale + neighborKey);

                                        hubKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));

hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
                                        puts.add(hubKeyPut);

                                        groupKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));

groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
Bytes.toBytes(groupNeighborEntry.getKey()));
                                        puts.add(groupKeyPut);

                                        topGroupKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));

topGroupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TOP_GROUP_KEY_COLUMN),
Bytes.toBytes(GroupRegistry.WWW().GetParentGroupKey(groupNeighborEntry.getKey())));
                                        puts.add(topGroupKeyPut);

                                        timingScalePut = new
Put(Bytes.toBytes(hhNeighborRowKey));

timingScalePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TIMING_SCALE_COLUMN),
Bytes.toBytes(timingScale));
                                        puts.add(timingScalePut);

                                        nodeKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));

nodeKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_NODE_KEY_COLUMN),
Bytes.toBytes(neighborKey));
                                        puts.add(nodeKeyPut);

                                        hubNeighborTypePut = new
Put(Bytes.toBytes(hhNeighborRowKey));

hubNeighborTypePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TYPE_COLUMN),
Bytes.toBytes(SocialRole.VIRTUAL_NEIGHBOR));
                                        puts.add(hubNeighborTypePut);
                                }
                        }
                }

                try
                {
                        this.neighborTable.put(puts);
                }
                catch (IOException e)
                {
                        e.printStackTrace();
                }
        }
        ......

Thanks so much!

Best regards,
Bing