Bing Li
2012-08-29, 14:04
Bing Li
2012-08-29, 14:08
N Keywal
2012-08-29, 15:22
Bing Li
2012-08-29, 15:42
Mohammad Tariq
2012-08-29, 15:45
Mohammad Tariq
2012-08-29, 15:47
N Keywal
2012-08-29, 15:59
Bing Li
2012-08-29, 16:07
Bing Li
2012-08-29, 16:45
Young Y Kim
2012-08-30, 08:44
-
HBase Is So Slow To Save Data?Bing Li 2012-08-29, 14:04
Dear all,
According to my experiences, it is very slow for HBase to save data? Am I
right?
For example, today I need to save data in a HashMap to HBase. It took about
more than three hours. However when saving the same HashMap in a file in
the text format with the redirected System.out, it took only 4.5 seconds!
Why is HBase so slow? It is indexing?
My code to save data in HBase is as follows. I think the code must be
correct.
......
public synchronized void
AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int timingScale)
{
List<Put> puts = new ArrayList<Put>();
String hhNeighborRowKey;
Put hubKeyPut;
Put groupKeyPut;
Put topGroupKeyPut;
Put timingScalePut;
Put nodeKeyPut;
Put hubNeighborTypePut;
for (Map.Entry<String, ConcurrentHashMap<String,
Set<String>>> sourceHubGroupNeighborEntry : hhOutNeighborMap.entrySet())
{
for (Map.Entry<String, Set<String>>
groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
{
for (String neighborKey :
groupNeighborEntry.getValue())
{
hhNeighborRowKey NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
groupNeighborEntry.getKey() + timingScale + neighborKey);
hubKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));
hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
puts.add(hubKeyPut);
groupKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));
groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
Bytes.toBytes(groupNeighborEntry.getKey()));
puts.add(groupKeyPut);
topGroupKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));
topGroupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TOP_GROUP_KEY_COLUMN),
Bytes.toBytes(GroupRegistry.WWW().GetParentGroupKey(groupNeighborEntry.getKey())));
puts.add(topGroupKeyPut);
timingScalePut = new
Put(Bytes.toBytes(hhNeighborRowKey));
timingScalePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TIMING_SCALE_COLUMN),
Bytes.toBytes(timingScale));
puts.add(timingScalePut);
nodeKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));
nodeKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_NODE_KEY_COLUMN),
Bytes.toBytes(neighborKey));
puts.add(nodeKeyPut);
hubNeighborTypePut = new
Put(Bytes.toBytes(hhNeighborRowKey));
hubNeighborTypePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TYPE_COLUMN),
Bytes.toBytes(SocialRole.VIRTUAL_NEIGHBOR));
puts.add(hubNeighborTypePut);
}
}
}
try
{
this.neighborTable.put(puts);
}
catch (IOException e)
{
e.printStackTrace();
}
}
......
Thanks so much!
Best regards,
Bing
According to my experiences, it is very slow for HBase to save data? Am I
right?
For example, today I need to save data in a HashMap to HBase. It took about
more than three hours. However when saving the same HashMap in a file in
the text format with the redirected System.out, it took only 4.5 seconds!
Why is HBase so slow? It is indexing?
My code to save data in HBase is as follows. I think the code must be
correct.
......
public synchronized void
AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int timingScale)
{
List<Put> puts = new ArrayList<Put>();
String hhNeighborRowKey;
Put hubKeyPut;
Put groupKeyPut;
Put topGroupKeyPut;
Put timingScalePut;
Put nodeKeyPut;
Put hubNeighborTypePut;
for (Map.Entry<String, ConcurrentHashMap<String,
Set<String>>> sourceHubGroupNeighborEntry : hhOutNeighborMap.entrySet())
{
for (Map.Entry<String, Set<String>>
groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
{
for (String neighborKey :
groupNeighborEntry.getValue())
{
hhNeighborRowKey NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
groupNeighborEntry.getKey() + timingScale + neighborKey);
hubKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));
hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
puts.add(hubKeyPut);
groupKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));
groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
Bytes.toBytes(groupNeighborEntry.getKey()));
puts.add(groupKeyPut);
topGroupKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));
topGroupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TOP_GROUP_KEY_COLUMN),
Bytes.toBytes(GroupRegistry.WWW().GetParentGroupKey(groupNeighborEntry.getKey())));
puts.add(topGroupKeyPut);
timingScalePut = new
Put(Bytes.toBytes(hhNeighborRowKey));
timingScalePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TIMING_SCALE_COLUMN),
Bytes.toBytes(timingScale));
puts.add(timingScalePut);
nodeKeyPut = new
Put(Bytes.toBytes(hhNeighborRowKey));
nodeKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_NODE_KEY_COLUMN),
Bytes.toBytes(neighborKey));
puts.add(nodeKeyPut);
hubNeighborTypePut = new
Put(Bytes.toBytes(hhNeighborRowKey));
hubNeighborTypePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TYPE_COLUMN),
Bytes.toBytes(SocialRole.VIRTUAL_NEIGHBOR));
puts.add(hubNeighborTypePut);
}
}
}
try
{
this.neighborTable.put(puts);
}
catch (IOException e)
{
e.printStackTrace();
}
}
......
Thanks so much!
Best regards,
Bing
-
Re: HBase Is So Slow To Save Data?Bing Li 2012-08-29, 14:08
Dear all,
By the way, my HBase is in the pseudo-distributed mode. Thanks!
Best regards,
Bing
On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear all,
>
> According to my experiences, it is very slow for HBase to save data? Am I
> right?
>
> For example, today I need to save data in a HashMap to HBase. It took
> about more than three hours. However when saving the same HashMap in a file
> in the text format with the redirected System.out, it took only 4.5 seconds!
>
> Why is HBase so slow? It is indexing?
>
> My code to save data in HBase is as follows. I think the code must be
> correct.
>
> ......
> public synchronized void
> AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
> ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int timingScale)
> {
> List<Put> puts = new ArrayList<Put>();
>
> String hhNeighborRowKey;
> Put hubKeyPut;
> Put groupKeyPut;
> Put topGroupKeyPut;
> Put timingScalePut;
> Put nodeKeyPut;
> Put hubNeighborTypePut;
>
> for (Map.Entry<String, ConcurrentHashMap<String,
> Set<String>>> sourceHubGroupNeighborEntry : hhOutNeighborMap.entrySet())
> {
> for (Map.Entry<String, Set<String>>
> groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
> {
> for (String neighborKey :
> groupNeighborEntry.getValue())
> {
> hhNeighborRowKey > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
> Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
> groupNeighborEntry.getKey() + timingScale + neighborKey);
>
> hubKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
> Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
> puts.add(hubKeyPut);
>
> groupKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
> Bytes.toBytes(groupNeighborEntry.getKey()));
> puts.add(groupKeyPut);
>
> topGroupKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> topGroupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TOP_GROUP_KEY_COLUMN),
> Bytes.toBytes(GroupRegistry.WWW().GetParentGroupKey(groupNeighborEntry.getKey())));
> puts.add(topGroupKeyPut);
>
> timingScalePut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> timingScalePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TIMING_SCALE_COLUMN),
> Bytes.toBytes(timingScale));
> puts.add(timingScalePut);
>
> nodeKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> nodeKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_NODE_KEY_COLUMN),
> Bytes.toBytes(neighborKey));
> puts.add(nodeKeyPut);
>
> hubNeighborTypePut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> hubNeighborTypePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TYPE_COLUMN),
> Bytes.toBytes(SocialRole.VIRTUAL_NEIGHBOR));
> puts.add(hubNeighborTypePut);
By the way, my HBase is in the pseudo-distributed mode. Thanks!
Best regards,
Bing
On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear all,
>
> According to my experiences, it is very slow for HBase to save data? Am I
> right?
>
> For example, today I need to save data in a HashMap to HBase. It took
> about more than three hours. However when saving the same HashMap in a file
> in the text format with the redirected System.out, it took only 4.5 seconds!
>
> Why is HBase so slow? It is indexing?
>
> My code to save data in HBase is as follows. I think the code must be
> correct.
>
> ......
> public synchronized void
> AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
> ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int timingScale)
> {
> List<Put> puts = new ArrayList<Put>();
>
> String hhNeighborRowKey;
> Put hubKeyPut;
> Put groupKeyPut;
> Put topGroupKeyPut;
> Put timingScalePut;
> Put nodeKeyPut;
> Put hubNeighborTypePut;
>
> for (Map.Entry<String, ConcurrentHashMap<String,
> Set<String>>> sourceHubGroupNeighborEntry : hhOutNeighborMap.entrySet())
> {
> for (Map.Entry<String, Set<String>>
> groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
> {
> for (String neighborKey :
> groupNeighborEntry.getValue())
> {
> hhNeighborRowKey > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
> Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
> groupNeighborEntry.getKey() + timingScale + neighborKey);
>
> hubKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
> Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
> puts.add(hubKeyPut);
>
> groupKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
> Bytes.toBytes(groupNeighborEntry.getKey()));
> puts.add(groupKeyPut);
>
> topGroupKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> topGroupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TOP_GROUP_KEY_COLUMN),
> Bytes.toBytes(GroupRegistry.WWW().GetParentGroupKey(groupNeighborEntry.getKey())));
> puts.add(topGroupKeyPut);
>
> timingScalePut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> timingScalePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TIMING_SCALE_COLUMN),
> Bytes.toBytes(timingScale));
> puts.add(timingScalePut);
>
> nodeKeyPut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> nodeKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_NODE_KEY_COLUMN),
> Bytes.toBytes(neighborKey));
> puts.add(nodeKeyPut);
>
> hubNeighborTypePut = new
> Put(Bytes.toBytes(hhNeighborRowKey));
>
> hubNeighborTypePut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TYPE_COLUMN),
> Bytes.toBytes(SocialRole.VIRTUAL_NEIGHBOR));
> puts.add(hubNeighborTypePut);
-
Re: HBase Is So Slow To Save Data?N Keywal 2012-08-29, 15:22
Hi Bing,
You should expect HBase to be slower in the generic case:
1) it writes much more data (see hbase data model), with extra columns
qualifiers, timestamps & so on.
2) the data is written multiple times: once in the write-ahead-log, once
per replica on datanode & so on again.
3) there are inter process calls & inter machine calls on the critical path.
This is the cost of the atomicity, reliability and scalability features.
With these features in mind, HBase is reasonably fast to save data on a
cluster.
On your specific case (without the points 2 & 3 above), the performance
seems to be very bad.
You should first look at:
- how much is spent in the put vs. preparing the list
- do you have garbage collection going on? even swap?
- what's the size of your final Array vs. the available memory?
Cheers,
N.
On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear all,
>
> By the way, my HBase is in the pseudo-distributed mode. Thanks!
>
> Best regards,
> Bing
>
> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>
> > Dear all,
> >
> > According to my experiences, it is very slow for HBase to save data? Am I
> > right?
> >
> > For example, today I need to save data in a HashMap to HBase. It took
> > about more than three hours. However when saving the same HashMap in a
> file
> > in the text format with the redirected System.out, it took only 4.5
> seconds!
> >
> > Why is HBase so slow? It is indexing?
> >
> > My code to save data in HBase is as follows. I think the code must be
> > correct.
> >
> > ......
> > public synchronized void
> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
> timingScale)
> > {
> > List<Put> puts = new ArrayList<Put>();
> >
> > String hhNeighborRowKey;
> > Put hubKeyPut;
> > Put groupKeyPut;
> > Put topGroupKeyPut;
> > Put timingScalePut;
> > Put nodeKeyPut;
> > Put hubNeighborTypePut;
> >
> > for (Map.Entry<String, ConcurrentHashMap<String,
> > Set<String>>> sourceHubGroupNeighborEntry : hhOutNeighborMap.entrySet())
> > {
> > for (Map.Entry<String, Set<String>>
> > groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
> > {
> > for (String neighborKey :
> > groupNeighborEntry.getValue())
> > {
> > hhNeighborRowKey > > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
> > Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
> > groupNeighborEntry.getKey() + timingScale + neighborKey);
> >
> > hubKeyPut = new
> > Put(Bytes.toBytes(hhNeighborRowKey));
> >
> > hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
> > Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
> > puts.add(hubKeyPut);
> >
> > groupKeyPut = new
> > Put(Bytes.toBytes(hhNeighborRowKey));
> >
> > groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
> > Bytes.toBytes(groupNeighborEntry.getKey()));
> > puts.add(groupKeyPut);
> >
> > topGroupKeyPut = new
> > Put(Bytes.toBytes(hhNeighborRowKey));
> >
> >
> topGroupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TOP_GROUP_KEY_COLUMN),
> >
> Bytes.toBytes(GroupRegistry.WWW().GetParentGroupKey(groupNeighborEntry.getKey())));
> > puts.add(topGroupKeyPut);
You should expect HBase to be slower in the generic case:
1) it writes much more data (see hbase data model), with extra columns
qualifiers, timestamps & so on.
2) the data is written multiple times: once in the write-ahead-log, once
per replica on datanode & so on again.
3) there are inter process calls & inter machine calls on the critical path.
This is the cost of the atomicity, reliability and scalability features.
With these features in mind, HBase is reasonably fast to save data on a
cluster.
On your specific case (without the points 2 & 3 above), the performance
seems to be very bad.
You should first look at:
- how much is spent in the put vs. preparing the list
- do you have garbage collection going on? even swap?
- what's the size of your final Array vs. the available memory?
Cheers,
N.
On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear all,
>
> By the way, my HBase is in the pseudo-distributed mode. Thanks!
>
> Best regards,
> Bing
>
> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>
> > Dear all,
> >
> > According to my experiences, it is very slow for HBase to save data? Am I
> > right?
> >
> > For example, today I need to save data in a HashMap to HBase. It took
> > about more than three hours. However when saving the same HashMap in a
> file
> > in the text format with the redirected System.out, it took only 4.5
> seconds!
> >
> > Why is HBase so slow? It is indexing?
> >
> > My code to save data in HBase is as follows. I think the code must be
> > correct.
> >
> > ......
> > public synchronized void
> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
> timingScale)
> > {
> > List<Put> puts = new ArrayList<Put>();
> >
> > String hhNeighborRowKey;
> > Put hubKeyPut;
> > Put groupKeyPut;
> > Put topGroupKeyPut;
> > Put timingScalePut;
> > Put nodeKeyPut;
> > Put hubNeighborTypePut;
> >
> > for (Map.Entry<String, ConcurrentHashMap<String,
> > Set<String>>> sourceHubGroupNeighborEntry : hhOutNeighborMap.entrySet())
> > {
> > for (Map.Entry<String, Set<String>>
> > groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
> > {
> > for (String neighborKey :
> > groupNeighborEntry.getValue())
> > {
> > hhNeighborRowKey > > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
> > Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
> > groupNeighborEntry.getKey() + timingScale + neighborKey);
> >
> > hubKeyPut = new
> > Put(Bytes.toBytes(hhNeighborRowKey));
> >
> > hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
> > Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
> > puts.add(hubKeyPut);
> >
> > groupKeyPut = new
> > Put(Bytes.toBytes(hhNeighborRowKey));
> >
> > groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
> > Bytes.toBytes(groupNeighborEntry.getKey()));
> > puts.add(groupKeyPut);
> >
> > topGroupKeyPut = new
> > Put(Bytes.toBytes(hhNeighborRowKey));
> >
> >
> topGroupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_TOP_GROUP_KEY_COLUMN),
> >
> Bytes.toBytes(GroupRegistry.WWW().GetParentGroupKey(groupNeighborEntry.getKey())));
> > puts.add(topGroupKeyPut);
-
Re: HBase Is So Slow To Save Data?Bing Li 2012-08-29, 15:42
Dear N Keywal,
Thanks so much for your reply!
The total amount of data is about 110M. The available memory is enough, 2G.
In Java, I just set a collection to NULL to collect garbage. Do you think
it is fine?
Best regards,
Bing
On Wed, Aug 29, 2012 at 11:22 PM, N Keywal <[EMAIL PROTECTED]> wrote:
> Hi Bing,
>
> You should expect HBase to be slower in the generic case:
> 1) it writes much more data (see hbase data model), with extra columns
> qualifiers, timestamps & so on.
> 2) the data is written multiple times: once in the write-ahead-log, once
> per replica on datanode & so on again.
> 3) there are inter process calls & inter machine calls on the critical
> path.
>
> This is the cost of the atomicity, reliability and scalability features.
> With these features in mind, HBase is reasonably fast to save data on a
> cluster.
>
> On your specific case (without the points 2 & 3 above), the performance
> seems to be very bad.
>
> You should first look at:
> - how much is spent in the put vs. preparing the list
> - do you have garbage collection going on? even swap?
> - what's the size of your final Array vs. the available memory?
>
> Cheers,
>
> N.
>
>
>
> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>
>> Dear all,
>>
>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
>>
>> Best regards,
>> Bing
>>
>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>
>> > Dear all,
>> >
>> > According to my experiences, it is very slow for HBase to save data? Am
>> I
>> > right?
>> >
>> > For example, today I need to save data in a HashMap to HBase. It took
>> > about more than three hours. However when saving the same HashMap in a
>> file
>> > in the text format with the redirected System.out, it took only 4.5
>> seconds!
>> >
>> > Why is HBase so slow? It is indexing?
>> >
>> > My code to save data in HBase is as follows. I think the code must be
>> > correct.
>> >
>> > ......
>> > public synchronized void
>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
>> timingScale)
>> > {
>> > List<Put> puts = new ArrayList<Put>();
>> >
>> > String hhNeighborRowKey;
>> > Put hubKeyPut;
>> > Put groupKeyPut;
>> > Put topGroupKeyPut;
>> > Put timingScalePut;
>> > Put nodeKeyPut;
>> > Put hubNeighborTypePut;
>> >
>> > for (Map.Entry<String, ConcurrentHashMap<String,
>> > Set<String>>> sourceHubGroupNeighborEntry : hhOutNeighborMap.entrySet())
>> > {
>> > for (Map.Entry<String, Set<String>>
>> > groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
>> > {
>> > for (String neighborKey :
>> > groupNeighborEntry.getValue())
>> > {
>> > hhNeighborRowKey >> > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
>> > Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
>> > groupNeighborEntry.getKey() + timingScale + neighborKey);
>> >
>> > hubKeyPut = new
>> > Put(Bytes.toBytes(hhNeighborRowKey));
>> >
>> > hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
>> > Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
>> > puts.add(hubKeyPut);
>> >
>> > groupKeyPut = new
>> > Put(Bytes.toBytes(hhNeighborRowKey));
>> >
>> >
>> groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
>> > Bytes.toBytes(groupNeighborEntry.getKey()));
>> > puts.add(groupKeyPut);
Thanks so much for your reply!
The total amount of data is about 110M. The available memory is enough, 2G.
In Java, I just set a collection to NULL to collect garbage. Do you think
it is fine?
Best regards,
Bing
On Wed, Aug 29, 2012 at 11:22 PM, N Keywal <[EMAIL PROTECTED]> wrote:
> Hi Bing,
>
> You should expect HBase to be slower in the generic case:
> 1) it writes much more data (see hbase data model), with extra columns
> qualifiers, timestamps & so on.
> 2) the data is written multiple times: once in the write-ahead-log, once
> per replica on datanode & so on again.
> 3) there are inter process calls & inter machine calls on the critical
> path.
>
> This is the cost of the atomicity, reliability and scalability features.
> With these features in mind, HBase is reasonably fast to save data on a
> cluster.
>
> On your specific case (without the points 2 & 3 above), the performance
> seems to be very bad.
>
> You should first look at:
> - how much is spent in the put vs. preparing the list
> - do you have garbage collection going on? even swap?
> - what's the size of your final Array vs. the available memory?
>
> Cheers,
>
> N.
>
>
>
> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>
>> Dear all,
>>
>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
>>
>> Best regards,
>> Bing
>>
>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>
>> > Dear all,
>> >
>> > According to my experiences, it is very slow for HBase to save data? Am
>> I
>> > right?
>> >
>> > For example, today I need to save data in a HashMap to HBase. It took
>> > about more than three hours. However when saving the same HashMap in a
>> file
>> > in the text format with the redirected System.out, it took only 4.5
>> seconds!
>> >
>> > Why is HBase so slow? It is indexing?
>> >
>> > My code to save data in HBase is as follows. I think the code must be
>> > correct.
>> >
>> > ......
>> > public synchronized void
>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
>> timingScale)
>> > {
>> > List<Put> puts = new ArrayList<Put>();
>> >
>> > String hhNeighborRowKey;
>> > Put hubKeyPut;
>> > Put groupKeyPut;
>> > Put topGroupKeyPut;
>> > Put timingScalePut;
>> > Put nodeKeyPut;
>> > Put hubNeighborTypePut;
>> >
>> > for (Map.Entry<String, ConcurrentHashMap<String,
>> > Set<String>>> sourceHubGroupNeighborEntry : hhOutNeighborMap.entrySet())
>> > {
>> > for (Map.Entry<String, Set<String>>
>> > groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
>> > {
>> > for (String neighborKey :
>> > groupNeighborEntry.getValue())
>> > {
>> > hhNeighborRowKey >> > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
>> > Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
>> > groupNeighborEntry.getKey() + timingScale + neighborKey);
>> >
>> > hubKeyPut = new
>> > Put(Bytes.toBytes(hhNeighborRowKey));
>> >
>> > hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
>> > Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
>> > puts.add(hubKeyPut);
>> >
>> > groupKeyPut = new
>> > Put(Bytes.toBytes(hhNeighborRowKey));
>> >
>> >
>> groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
>> > Bytes.toBytes(groupNeighborEntry.getKey()));
>> > puts.add(groupKeyPut);
-
Re: HBase Is So Slow To Save Data?Mohammad Tariq 2012-08-29, 15:45
Pseudo-distributed setup could be a cause.
On Wednesday, August 29, 2012, N Keywal <[EMAIL PROTECTED]> wrote:
> Hi Bing,
>
> You should expect HBase to be slower in the generic case:
> 1) it writes much more data (see hbase data model), with extra columns
> qualifiers, timestamps & so on.
> 2) the data is written multiple times: once in the write-ahead-log, once
> per replica on datanode & so on again.
> 3) there are inter process calls & inter machine calls on the critical
path.
>
> This is the cost of the atomicity, reliability and scalability features.
> With these features in mind, HBase is reasonably fast to save data on a
> cluster.
>
> On your specific case (without the points 2 & 3 above), the performance
> seems to be very bad.
>
> You should first look at:
> - how much is spent in the put vs. preparing the list
> - do you have garbage collection going on? even swap?
> - what's the size of your final Array vs. the available memory?
>
> Cheers,
>
> N.
>
>
> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>
>> Dear all,
>>
>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
>>
>> Best regards,
>> Bing
>>
>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>
>> > Dear all,
>> >
>> > According to my experiences, it is very slow for HBase to save data?
Am I
>> > right?
>> >
>> > For example, today I need to save data in a HashMap to HBase. It took
>> > about more than three hours. However when saving the same HashMap in a
>> file
>> > in the text format with the redirected System.out, it took only 4.5
>> seconds!
>> >
>> > Why is HBase so slow? It is indexing?
>> >
>> > My code to save data in HBase is as follows. I think the code must be
>> > correct.
>> >
>> > ......
>> > public synchronized void
>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
>> timingScale)
>> > {
>> > List<Put> puts = new ArrayList<Put>();
>> >
>> > String hhNeighborRowKey;
>> > Put hubKeyPut;
>> > Put groupKeyPut;
>> > Put topGroupKeyPut;
>> > Put timingScalePut;
>> > Put nodeKeyPut;
>> > Put hubNeighborTypePut;
>> >
>> > for (Map.Entry<String, ConcurrentHashMap<String,
>> > Set<String>>> sourceHubGroupNeighborEntry :
hhOutNeighborMap.entrySet())
>> > {
>> > for (Map.Entry<String, Set<String>>
>> > groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
>> > {
>> > for (String neighborKey :
>> > groupNeighborEntry.getValue())
>> > {
>> > hhNeighborRowKey >> > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
>> > Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
>> > groupNeighborEntry.getKey() + timingScale + neighborKey);
>> >
>> > hubKeyPut = new
>> > Put(Bytes.toBytes(hhNeighborRowKey));
>> >
>> > hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
>> > Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
>> > puts.add(hubKeyPut);
>> >
>> > groupKeyPut = new
>> > Put(Bytes.toBytes(hhNeighborRowKey));
>> >
>> >
groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
>> > Bytes.toBytes(groupNeighborEntry.getKey()));
>> >
--
Regards,
Mohammad Tariq
On Wednesday, August 29, 2012, N Keywal <[EMAIL PROTECTED]> wrote:
> Hi Bing,
>
> You should expect HBase to be slower in the generic case:
> 1) it writes much more data (see hbase data model), with extra columns
> qualifiers, timestamps & so on.
> 2) the data is written multiple times: once in the write-ahead-log, once
> per replica on datanode & so on again.
> 3) there are inter process calls & inter machine calls on the critical
path.
>
> This is the cost of the atomicity, reliability and scalability features.
> With these features in mind, HBase is reasonably fast to save data on a
> cluster.
>
> On your specific case (without the points 2 & 3 above), the performance
> seems to be very bad.
>
> You should first look at:
> - how much is spent in the put vs. preparing the list
> - do you have garbage collection going on? even swap?
> - what's the size of your final Array vs. the available memory?
>
> Cheers,
>
> N.
>
>
> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>
>> Dear all,
>>
>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
>>
>> Best regards,
>> Bing
>>
>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>
>> > Dear all,
>> >
>> > According to my experiences, it is very slow for HBase to save data?
Am I
>> > right?
>> >
>> > For example, today I need to save data in a HashMap to HBase. It took
>> > about more than three hours. However when saving the same HashMap in a
>> file
>> > in the text format with the redirected System.out, it took only 4.5
>> seconds!
>> >
>> > Why is HBase so slow? It is indexing?
>> >
>> > My code to save data in HBase is as follows. I think the code must be
>> > correct.
>> >
>> > ......
>> > public synchronized void
>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
>> timingScale)
>> > {
>> > List<Put> puts = new ArrayList<Put>();
>> >
>> > String hhNeighborRowKey;
>> > Put hubKeyPut;
>> > Put groupKeyPut;
>> > Put topGroupKeyPut;
>> > Put timingScalePut;
>> > Put nodeKeyPut;
>> > Put hubNeighborTypePut;
>> >
>> > for (Map.Entry<String, ConcurrentHashMap<String,
>> > Set<String>>> sourceHubGroupNeighborEntry :
hhOutNeighborMap.entrySet())
>> > {
>> > for (Map.Entry<String, Set<String>>
>> > groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
>> > {
>> > for (String neighborKey :
>> > groupNeighborEntry.getValue())
>> > {
>> > hhNeighborRowKey >> > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
>> > Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
>> > groupNeighborEntry.getKey() + timingScale + neighborKey);
>> >
>> > hubKeyPut = new
>> > Put(Bytes.toBytes(hhNeighborRowKey));
>> >
>> > hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
>> > Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
>> > puts.add(hubKeyPut);
>> >
>> > groupKeyPut = new
>> > Put(Bytes.toBytes(hhNeighborRowKey));
>> >
>> >
groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
>> > Bytes.toBytes(groupNeighborEntry.getKey()));
>> >
--
Regards,
Mohammad Tariq
-
Re: HBase Is So Slow To Save Data?Mohammad Tariq 2012-08-29, 15:47
I don't think 2G is sufficient enough keeping in mind that all the hadoop
daemons are running on the same box. (Maybe your IDE and other stuff too).
On Wednesday, August 29, 2012, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Pseudo-distributed setup could be a cause.
>
> On Wednesday, August 29, 2012, N Keywal <[EMAIL PROTECTED]> wrote:
>> Hi Bing,
>>
>> You should expect HBase to be slower in the generic case:
>> 1) it writes much more data (see hbase data model), with extra columns
>> qualifiers, timestamps & so on.
>> 2) the data is written multiple times: once in the write-ahead-log, once
>> per replica on datanode & so on again.
>> 3) there are inter process calls & inter machine calls on the critical
path.
>>
>> This is the cost of the atomicity, reliability and scalability features.
>> With these features in mind, HBase is reasonably fast to save data on a
>> cluster.
>>
>> On your specific case (without the points 2 & 3 above), the performance
>> seems to be very bad.
>>
>> You should first look at:
>> - how much is spent in the put vs. preparing the list
>> - do you have garbage collection going on? even swap?
>> - what's the size of your final Array vs. the available memory?
>>
>> Cheers,
>>
>> N.
>>
>>
>> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>
>>> Dear all,
>>>
>>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
>>>
>>> Best regards,
>>> Bing
>>>
>>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>>
>>> > Dear all,
>>> >
>>> > According to my experiences, it is very slow for HBase to save data?
Am I
>>> > right?
>>> >
>>> > For example, today I need to save data in a HashMap to HBase. It took
>>> > about more than three hours. However when saving the same HashMap in a
>>> file
>>> > in the text format with the redirected System.out, it took only 4.5
>>> seconds!
>>> >
>>> > Why is HBase so slow? It is indexing?
>>> >
>>> > My code to save data in HBase is as follows. I think the code must be
>>> > correct.
>>> >
>>> > ......
>>> > public synchronized void
>>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
>>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
>>> timingScale)
>>> > {
>>> > List<Put> puts = new ArrayList<Put>();
>>> >
>>> > String hhNeighborRowKey;
>>> > Put hubKeyPut;
>>> > Put groupKeyPut;
>>> > Put topGroupKeyPut;
>>> > Put timingScalePut;
>>> > Put nodeKeyPut;
>>> > Put hubNeighborTypePut;
>>> >
>>> > for (Map.Entry<String, ConcurrentHashMap<String,
>>> > Set<String>>> sourceHubGroupNeighborEntry :
hhOutNeighborMap.entrySet())
>>> > {
>>> > for (Map.Entry<String, Set<String>>
>>> > groupNeighborEntry :
sourceHubGroupNeighborEntry.getValue().entrySet())
>>> > {
>>> > for (String neighborKey :
>>> > groupNeighborEntry.getValue())
>>> > {
>>> > hhNeighborRowKey >>> > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
>>> > Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
>>> > groupNeighborEntry.getKey() + timingScale + neighborKey);
>>> >
>>> > hubKeyPut = new
>>> > Put(Bytes.toBytes(hhNeighborRowKey));
>>> >
>>> >
hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
>>> > Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
>>> > puts.add(hubKeyPut);
>>> >
>>> > groupKeyPut = new
>>> > Put(Bytes.toBytes(hhNeighborRowKey));
>>> >
>>> >
groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
Regards,
Mohammad Tariq
daemons are running on the same box. (Maybe your IDE and other stuff too).
On Wednesday, August 29, 2012, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
> Pseudo-distributed setup could be a cause.
>
> On Wednesday, August 29, 2012, N Keywal <[EMAIL PROTECTED]> wrote:
>> Hi Bing,
>>
>> You should expect HBase to be slower in the generic case:
>> 1) it writes much more data (see hbase data model), with extra columns
>> qualifiers, timestamps & so on.
>> 2) the data is written multiple times: once in the write-ahead-log, once
>> per replica on datanode & so on again.
>> 3) there are inter process calls & inter machine calls on the critical
path.
>>
>> This is the cost of the atomicity, reliability and scalability features.
>> With these features in mind, HBase is reasonably fast to save data on a
>> cluster.
>>
>> On your specific case (without the points 2 & 3 above), the performance
>> seems to be very bad.
>>
>> You should first look at:
>> - how much is spent in the put vs. preparing the list
>> - do you have garbage collection going on? even swap?
>> - what's the size of your final Array vs. the available memory?
>>
>> Cheers,
>>
>> N.
>>
>>
>> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>
>>> Dear all,
>>>
>>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
>>>
>>> Best regards,
>>> Bing
>>>
>>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>>
>>> > Dear all,
>>> >
>>> > According to my experiences, it is very slow for HBase to save data?
Am I
>>> > right?
>>> >
>>> > For example, today I need to save data in a HashMap to HBase. It took
>>> > about more than three hours. However when saving the same HashMap in a
>>> file
>>> > in the text format with the redirected System.out, it took only 4.5
>>> seconds!
>>> >
>>> > Why is HBase so slow? It is indexing?
>>> >
>>> > My code to save data in HBase is as follows. I think the code must be
>>> > correct.
>>> >
>>> > ......
>>> > public synchronized void
>>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
>>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
>>> timingScale)
>>> > {
>>> > List<Put> puts = new ArrayList<Put>();
>>> >
>>> > String hhNeighborRowKey;
>>> > Put hubKeyPut;
>>> > Put groupKeyPut;
>>> > Put topGroupKeyPut;
>>> > Put timingScalePut;
>>> > Put nodeKeyPut;
>>> > Put hubNeighborTypePut;
>>> >
>>> > for (Map.Entry<String, ConcurrentHashMap<String,
>>> > Set<String>>> sourceHubGroupNeighborEntry :
hhOutNeighborMap.entrySet())
>>> > {
>>> > for (Map.Entry<String, Set<String>>
>>> > groupNeighborEntry :
sourceHubGroupNeighborEntry.getValue().entrySet())
>>> > {
>>> > for (String neighborKey :
>>> > groupNeighborEntry.getValue())
>>> > {
>>> > hhNeighborRowKey >>> > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
>>> > Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
>>> > groupNeighborEntry.getKey() + timingScale + neighborKey);
>>> >
>>> > hubKeyPut = new
>>> > Put(Bytes.toBytes(hhNeighborRowKey));
>>> >
>>> >
hubKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_HUB_KEY_COLUMN),
>>> > Bytes.toBytes(sourceHubGroupNeighborEntry.getKey()));
>>> > puts.add(hubKeyPut);
>>> >
>>> > groupKeyPut = new
>>> > Put(Bytes.toBytes(hhNeighborRowKey));
>>> >
>>> >
groupKeyPut.add(Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_FAMILY),
>>> > Bytes.toBytes(NeighborStructure.HUB_HUB_NEIGHBOR_GROUP_KEY_COLUMN),
Regards,
Mohammad Tariq
-
Re: HBase Is So Slow To Save Data?N Keywal 2012-08-29, 15:59
It's not useful here: if you have a memory issue, it's when your using the
list, not when you have finished with it and set it to null.
You need to monitor the memory consumption of the jvm, both the client &
the server.
Google around these keywords, there are many examples on the web.
Google as well arrayList initialization.
Note as well that the important is not the memory size of the structure on
disk but the size of the" List<Put> puts = new ArrayList<Put>();" before
the table put.
On Wed, Aug 29, 2012 at 5:42 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear N Keywal,
>
> Thanks so much for your reply!
>
> The total amount of data is about 110M. The available memory is enough, 2G.
>
> In Java, I just set a collection to NULL to collect garbage. Do you think
> it is fine?
>
> Best regards,
> Bing
>
>
> On Wed, Aug 29, 2012 at 11:22 PM, N Keywal <[EMAIL PROTECTED]> wrote:
>
>> Hi Bing,
>>
>> You should expect HBase to be slower in the generic case:
>> 1) it writes much more data (see hbase data model), with extra columns
>> qualifiers, timestamps & so on.
>> 2) the data is written multiple times: once in the write-ahead-log, once
>> per replica on datanode & so on again.
>> 3) there are inter process calls & inter machine calls on the critical
>> path.
>>
>> This is the cost of the atomicity, reliability and scalability features.
>> With these features in mind, HBase is reasonably fast to save data on a
>> cluster.
>>
>> On your specific case (without the points 2 & 3 above), the performance
>> seems to be very bad.
>>
>> You should first look at:
>> - how much is spent in the put vs. preparing the list
>> - do you have garbage collection going on? even swap?
>> - what's the size of your final Array vs. the available memory?
>>
>> Cheers,
>>
>> N.
>>
>>
>>
>> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>
>>> Dear all,
>>>
>>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
>>>
>>> Best regards,
>>> Bing
>>>
>>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>>
>>> > Dear all,
>>> >
>>> > According to my experiences, it is very slow for HBase to save data?
>>> Am I
>>> > right?
>>> >
>>> > For example, today I need to save data in a HashMap to HBase. It took
>>> > about more than three hours. However when saving the same HashMap in a
>>> file
>>> > in the text format with the redirected System.out, it took only 4.5
>>> seconds!
>>> >
>>> > Why is HBase so slow? It is indexing?
>>> >
>>> > My code to save data in HBase is as follows. I think the code must be
>>> > correct.
>>> >
>>> > ......
>>> > public synchronized void
>>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
>>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
>>> timingScale)
>>> > {
>>> > List<Put> puts = new ArrayList<Put>();
>>> >
>>> > String hhNeighborRowKey;
>>> > Put hubKeyPut;
>>> > Put groupKeyPut;
>>> > Put topGroupKeyPut;
>>> > Put timingScalePut;
>>> > Put nodeKeyPut;
>>> > Put hubNeighborTypePut;
>>> >
>>> > for (Map.Entry<String, ConcurrentHashMap<String,
>>> > Set<String>>> sourceHubGroupNeighborEntry :
>>> hhOutNeighborMap.entrySet())
>>> > {
>>> > for (Map.Entry<String, Set<String>>
>>> > groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
>>> > {
>>> > for (String neighborKey :
>>> > groupNeighborEntry.getValue())
>>> > {
>>> > hhNeighborRowKey >>> > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
>>> > Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
>>> > groupNeighborEntry.getKey() + timingScale + neighborKey);
>>> >
>>> > hubKeyPut = new
>>> > Put(Bytes.toBytes(hhNeighborRowKey));
list, not when you have finished with it and set it to null.
You need to monitor the memory consumption of the jvm, both the client &
the server.
Google around these keywords, there are many examples on the web.
Google as well arrayList initialization.
Note as well that the important is not the memory size of the structure on
disk but the size of the" List<Put> puts = new ArrayList<Put>();" before
the table put.
On Wed, Aug 29, 2012 at 5:42 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear N Keywal,
>
> Thanks so much for your reply!
>
> The total amount of data is about 110M. The available memory is enough, 2G.
>
> In Java, I just set a collection to NULL to collect garbage. Do you think
> it is fine?
>
> Best regards,
> Bing
>
>
> On Wed, Aug 29, 2012 at 11:22 PM, N Keywal <[EMAIL PROTECTED]> wrote:
>
>> Hi Bing,
>>
>> You should expect HBase to be slower in the generic case:
>> 1) it writes much more data (see hbase data model), with extra columns
>> qualifiers, timestamps & so on.
>> 2) the data is written multiple times: once in the write-ahead-log, once
>> per replica on datanode & so on again.
>> 3) there are inter process calls & inter machine calls on the critical
>> path.
>>
>> This is the cost of the atomicity, reliability and scalability features.
>> With these features in mind, HBase is reasonably fast to save data on a
>> cluster.
>>
>> On your specific case (without the points 2 & 3 above), the performance
>> seems to be very bad.
>>
>> You should first look at:
>> - how much is spent in the put vs. preparing the list
>> - do you have garbage collection going on? even swap?
>> - what's the size of your final Array vs. the available memory?
>>
>> Cheers,
>>
>> N.
>>
>>
>>
>> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>
>>> Dear all,
>>>
>>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
>>>
>>> Best regards,
>>> Bing
>>>
>>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>>>
>>> > Dear all,
>>> >
>>> > According to my experiences, it is very slow for HBase to save data?
>>> Am I
>>> > right?
>>> >
>>> > For example, today I need to save data in a HashMap to HBase. It took
>>> > about more than three hours. However when saving the same HashMap in a
>>> file
>>> > in the text format with the redirected System.out, it took only 4.5
>>> seconds!
>>> >
>>> > Why is HBase so slow? It is indexing?
>>> >
>>> > My code to save data in HBase is as follows. I think the code must be
>>> > correct.
>>> >
>>> > ......
>>> > public synchronized void
>>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
>>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
>>> timingScale)
>>> > {
>>> > List<Put> puts = new ArrayList<Put>();
>>> >
>>> > String hhNeighborRowKey;
>>> > Put hubKeyPut;
>>> > Put groupKeyPut;
>>> > Put topGroupKeyPut;
>>> > Put timingScalePut;
>>> > Put nodeKeyPut;
>>> > Put hubNeighborTypePut;
>>> >
>>> > for (Map.Entry<String, ConcurrentHashMap<String,
>>> > Set<String>>> sourceHubGroupNeighborEntry :
>>> hhOutNeighborMap.entrySet())
>>> > {
>>> > for (Map.Entry<String, Set<String>>
>>> > groupNeighborEntry : sourceHubGroupNeighborEntry.getValue().entrySet())
>>> > {
>>> > for (String neighborKey :
>>> > groupNeighborEntry.getValue())
>>> > {
>>> > hhNeighborRowKey >>> > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
>>> > Tools.GetAHash(sourceHubGroupNeighborEntry.getKey() +
>>> > groupNeighborEntry.getKey() + timingScale + neighborKey);
>>> >
>>> > hubKeyPut = new
>>> > Put(Bytes.toBytes(hhNeighborRowKey));
-
Re: HBase Is So Slow To Save Data?Bing Li 2012-08-29, 16:07
I see. Thanks so much!
Bing
On Wed, Aug 29, 2012 at 11:59 PM, N Keywal <[EMAIL PROTECTED]> wrote:
> It's not useful here: if you have a memory issue, it's when your using the
> list, not when you have finished with it and set it to null.
> You need to monitor the memory consumption of the jvm, both the client &
> the server.
> Google around these keywords, there are many examples on the web.
> Google as well arrayList initialization.
>
> Note as well that the important is not the memory size of the structure on
> disk but the size of the" List<Put> puts = new ArrayList<Put>();" before
> the table put.
>
> On Wed, Aug 29, 2012 at 5:42 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>
> > Dear N Keywal,
> >
> > Thanks so much for your reply!
> >
> > The total amount of data is about 110M. The available memory is enough,
> 2G.
> >
> > In Java, I just set a collection to NULL to collect garbage. Do you think
> > it is fine?
> >
> > Best regards,
> > Bing
> >
> >
> > On Wed, Aug 29, 2012 at 11:22 PM, N Keywal <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Bing,
> >>
> >> You should expect HBase to be slower in the generic case:
> >> 1) it writes much more data (see hbase data model), with extra columns
> >> qualifiers, timestamps & so on.
> >> 2) the data is written multiple times: once in the write-ahead-log, once
> >> per replica on datanode & so on again.
> >> 3) there are inter process calls & inter machine calls on the critical
> >> path.
> >>
> >> This is the cost of the atomicity, reliability and scalability features.
> >> With these features in mind, HBase is reasonably fast to save data on a
> >> cluster.
> >>
> >> On your specific case (without the points 2 & 3 above), the performance
> >> seems to be very bad.
> >>
> >> You should first look at:
> >> - how much is spent in the put vs. preparing the list
> >> - do you have garbage collection going on? even swap?
> >> - what's the size of your final Array vs. the available memory?
> >>
> >> Cheers,
> >>
> >> N.
> >>
> >>
> >>
> >> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> >>
> >>> Dear all,
> >>>
> >>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
> >>>
> >>> Best regards,
> >>> Bing
> >>>
> >>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> >>>
> >>> > Dear all,
> >>> >
> >>> > According to my experiences, it is very slow for HBase to save data?
> >>> Am I
> >>> > right?
> >>> >
> >>> > For example, today I need to save data in a HashMap to HBase. It took
> >>> > about more than three hours. However when saving the same HashMap in
> a
> >>> file
> >>> > in the text format with the redirected System.out, it took only 4.5
> >>> seconds!
> >>> >
> >>> > Why is HBase so slow? It is indexing?
> >>> >
> >>> > My code to save data in HBase is as follows. I think the code must be
> >>> > correct.
> >>> >
> >>> > ......
> >>> > public synchronized void
> >>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
> >>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
> >>> timingScale)
> >>> > {
> >>> > List<Put> puts = new ArrayList<Put>();
> >>> >
> >>> > String hhNeighborRowKey;
> >>> > Put hubKeyPut;
> >>> > Put groupKeyPut;
> >>> > Put topGroupKeyPut;
> >>> > Put timingScalePut;
> >>> > Put nodeKeyPut;
> >>> > Put hubNeighborTypePut;
> >>> >
> >>> > for (Map.Entry<String, ConcurrentHashMap<String,
> >>> > Set<String>>> sourceHubGroupNeighborEntry :
> >>> hhOutNeighborMap.entrySet())
> >>> > {
> >>> > for (Map.Entry<String, Set<String>>
> >>> > groupNeighborEntry :
> sourceHubGroupNeighborEntry.getValue().entrySet())
> >>> > {
> >>> > for (String neighborKey :
> >>> > groupNeighborEntry.getValue())
> >>> > {
> >>> > hhNeighborRowKey > >>> > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
Bing
On Wed, Aug 29, 2012 at 11:59 PM, N Keywal <[EMAIL PROTECTED]> wrote:
> It's not useful here: if you have a memory issue, it's when your using the
> list, not when you have finished with it and set it to null.
> You need to monitor the memory consumption of the jvm, both the client &
> the server.
> Google around these keywords, there are many examples on the web.
> Google as well arrayList initialization.
>
> Note as well that the important is not the memory size of the structure on
> disk but the size of the" List<Put> puts = new ArrayList<Put>();" before
> the table put.
>
> On Wed, Aug 29, 2012 at 5:42 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>
> > Dear N Keywal,
> >
> > Thanks so much for your reply!
> >
> > The total amount of data is about 110M. The available memory is enough,
> 2G.
> >
> > In Java, I just set a collection to NULL to collect garbage. Do you think
> > it is fine?
> >
> > Best regards,
> > Bing
> >
> >
> > On Wed, Aug 29, 2012 at 11:22 PM, N Keywal <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Bing,
> >>
> >> You should expect HBase to be slower in the generic case:
> >> 1) it writes much more data (see hbase data model), with extra columns
> >> qualifiers, timestamps & so on.
> >> 2) the data is written multiple times: once in the write-ahead-log, once
> >> per replica on datanode & so on again.
> >> 3) there are inter process calls & inter machine calls on the critical
> >> path.
> >>
> >> This is the cost of the atomicity, reliability and scalability features.
> >> With these features in mind, HBase is reasonably fast to save data on a
> >> cluster.
> >>
> >> On your specific case (without the points 2 & 3 above), the performance
> >> seems to be very bad.
> >>
> >> You should first look at:
> >> - how much is spent in the put vs. preparing the list
> >> - do you have garbage collection going on? even swap?
> >> - what's the size of your final Array vs. the available memory?
> >>
> >> Cheers,
> >>
> >> N.
> >>
> >>
> >>
> >> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> >>
> >>> Dear all,
> >>>
> >>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
> >>>
> >>> Best regards,
> >>> Bing
> >>>
> >>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> >>>
> >>> > Dear all,
> >>> >
> >>> > According to my experiences, it is very slow for HBase to save data?
> >>> Am I
> >>> > right?
> >>> >
> >>> > For example, today I need to save data in a HashMap to HBase. It took
> >>> > about more than three hours. However when saving the same HashMap in
> a
> >>> file
> >>> > in the text format with the redirected System.out, it took only 4.5
> >>> seconds!
> >>> >
> >>> > Why is HBase so slow? It is indexing?
> >>> >
> >>> > My code to save data in HBase is as follows. I think the code must be
> >>> > correct.
> >>> >
> >>> > ......
> >>> > public synchronized void
> >>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
> >>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
> >>> timingScale)
> >>> > {
> >>> > List<Put> puts = new ArrayList<Put>();
> >>> >
> >>> > String hhNeighborRowKey;
> >>> > Put hubKeyPut;
> >>> > Put groupKeyPut;
> >>> > Put topGroupKeyPut;
> >>> > Put timingScalePut;
> >>> > Put nodeKeyPut;
> >>> > Put hubNeighborTypePut;
> >>> >
> >>> > for (Map.Entry<String, ConcurrentHashMap<String,
> >>> > Set<String>>> sourceHubGroupNeighborEntry :
> >>> hhOutNeighborMap.entrySet())
> >>> > {
> >>> > for (Map.Entry<String, Set<String>>
> >>> > groupNeighborEntry :
> sourceHubGroupNeighborEntry.getValue().entrySet())
> >>> > {
> >>> > for (String neighborKey :
> >>> > groupNeighborEntry.getValue())
> >>> > {
> >>> > hhNeighborRowKey > >>> > NeighborStructure.HUB_HUB_NEIGHBOR_ROW +
-
Re: HBase Is So Slow To Save Data?Bing Li 2012-08-29, 16:45
Dear Cristofer,
Thanks so much for your reminding!
Best regards,
Bing
On Thu, Aug 30, 2012 at 12:32 AM, Cristofer Weber <
[EMAIL PROTECTED]> wrote:
> There's also a lot of conversions from same values to byte array
> representation, eg, your NeighborStructure constants. You should do this
> conversion only once to save time, since you are doing this inside 3 nested
> loops. Not sure about how much this can improve, but you should try this
> also.
>
> Best regards,
> Cristofer
>
> -----Mensagem original-----
> De: Bing Li [mailto:[EMAIL PROTECTED]]
> Enviada em: quarta-feira, 29 de agosto de 2012 13:07
> Para: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Assunto: Re: HBase Is So Slow To Save Data?
>
> I see. Thanks so much!
>
> Bing
>
>
> On Wed, Aug 29, 2012 at 11:59 PM, N Keywal <[EMAIL PROTECTED]> wrote:
>
> > It's not useful here: if you have a memory issue, it's when your using
> > the list, not when you have finished with it and set it to null.
> > You need to monitor the memory consumption of the jvm, both the client
> > & the server.
> > Google around these keywords, there are many examples on the web.
> > Google as well arrayList initialization.
> >
> > Note as well that the important is not the memory size of the
> > structure on disk but the size of the" List<Put> puts = new
> > ArrayList<Put>();" before the table put.
> >
> > On Wed, Aug 29, 2012 at 5:42 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> >
> > > Dear N Keywal,
> > >
> > > Thanks so much for your reply!
> > >
> > > The total amount of data is about 110M. The available memory is
> > > enough,
> > 2G.
> > >
> > > In Java, I just set a collection to NULL to collect garbage. Do you
> > > think it is fine?
> > >
> > > Best regards,
> > > Bing
> > >
> > >
> > > On Wed, Aug 29, 2012 at 11:22 PM, N Keywal <[EMAIL PROTECTED]> wrote:
> > >
> > >> Hi Bing,
> > >>
> > >> You should expect HBase to be slower in the generic case:
> > >> 1) it writes much more data (see hbase data model), with extra
> > >> columns qualifiers, timestamps & so on.
> > >> 2) the data is written multiple times: once in the write-ahead-log,
> > >> once per replica on datanode & so on again.
> > >> 3) there are inter process calls & inter machine calls on the
> > >> critical path.
> > >>
> > >> This is the cost of the atomicity, reliability and scalability
> features.
> > >> With these features in mind, HBase is reasonably fast to save data
> > >> on a cluster.
> > >>
> > >> On your specific case (without the points 2 & 3 above), the
> > >> performance seems to be very bad.
> > >>
> > >> You should first look at:
> > >> - how much is spent in the put vs. preparing the list
> > >> - do you have garbage collection going on? even swap?
> > >> - what's the size of your final Array vs. the available memory?
> > >>
> > >> Cheers,
> > >>
> > >> N.
> > >>
> > >>
> > >>
> > >> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > >>
> > >>> Dear all,
> > >>>
> > >>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
> > >>>
> > >>> Best regards,
> > >>> Bing
> > >>>
> > >>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > >>>
> > >>> > Dear all,
> > >>> >
> > >>> > According to my experiences, it is very slow for HBase to save
> data?
> > >>> Am I
> > >>> > right?
> > >>> >
> > >>> > For example, today I need to save data in a HashMap to HBase. It
> > >>> > took about more than three hours. However when saving the same
> > >>> > HashMap in
> > a
> > >>> file
> > >>> > in the text format with the redirected System.out, it took only
> > >>> > 4.5
> > >>> seconds!
> > >>> >
> > >>> > Why is HBase so slow? It is indexing?
> > >>> >
> > >>> > My code to save data in HBase is as follows. I think the code
> > >>> > must be correct.
> > >>> >
> > >>> > ......
> > >>> > public synchronized void
> > >>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
> > >>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
Thanks so much for your reminding!
Best regards,
Bing
On Thu, Aug 30, 2012 at 12:32 AM, Cristofer Weber <
[EMAIL PROTECTED]> wrote:
> There's also a lot of conversions from same values to byte array
> representation, eg, your NeighborStructure constants. You should do this
> conversion only once to save time, since you are doing this inside 3 nested
> loops. Not sure about how much this can improve, but you should try this
> also.
>
> Best regards,
> Cristofer
>
> -----Mensagem original-----
> De: Bing Li [mailto:[EMAIL PROTECTED]]
> Enviada em: quarta-feira, 29 de agosto de 2012 13:07
> Para: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Assunto: Re: HBase Is So Slow To Save Data?
>
> I see. Thanks so much!
>
> Bing
>
>
> On Wed, Aug 29, 2012 at 11:59 PM, N Keywal <[EMAIL PROTECTED]> wrote:
>
> > It's not useful here: if you have a memory issue, it's when your using
> > the list, not when you have finished with it and set it to null.
> > You need to monitor the memory consumption of the jvm, both the client
> > & the server.
> > Google around these keywords, there are many examples on the web.
> > Google as well arrayList initialization.
> >
> > Note as well that the important is not the memory size of the
> > structure on disk but the size of the" List<Put> puts = new
> > ArrayList<Put>();" before the table put.
> >
> > On Wed, Aug 29, 2012 at 5:42 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> >
> > > Dear N Keywal,
> > >
> > > Thanks so much for your reply!
> > >
> > > The total amount of data is about 110M. The available memory is
> > > enough,
> > 2G.
> > >
> > > In Java, I just set a collection to NULL to collect garbage. Do you
> > > think it is fine?
> > >
> > > Best regards,
> > > Bing
> > >
> > >
> > > On Wed, Aug 29, 2012 at 11:22 PM, N Keywal <[EMAIL PROTECTED]> wrote:
> > >
> > >> Hi Bing,
> > >>
> > >> You should expect HBase to be slower in the generic case:
> > >> 1) it writes much more data (see hbase data model), with extra
> > >> columns qualifiers, timestamps & so on.
> > >> 2) the data is written multiple times: once in the write-ahead-log,
> > >> once per replica on datanode & so on again.
> > >> 3) there are inter process calls & inter machine calls on the
> > >> critical path.
> > >>
> > >> This is the cost of the atomicity, reliability and scalability
> features.
> > >> With these features in mind, HBase is reasonably fast to save data
> > >> on a cluster.
> > >>
> > >> On your specific case (without the points 2 & 3 above), the
> > >> performance seems to be very bad.
> > >>
> > >> You should first look at:
> > >> - how much is spent in the put vs. preparing the list
> > >> - do you have garbage collection going on? even swap?
> > >> - what's the size of your final Array vs. the available memory?
> > >>
> > >> Cheers,
> > >>
> > >> N.
> > >>
> > >>
> > >>
> > >> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > >>
> > >>> Dear all,
> > >>>
> > >>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
> > >>>
> > >>> Best regards,
> > >>> Bing
> > >>>
> > >>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > >>>
> > >>> > Dear all,
> > >>> >
> > >>> > According to my experiences, it is very slow for HBase to save
> data?
> > >>> Am I
> > >>> > right?
> > >>> >
> > >>> > For example, today I need to save data in a HashMap to HBase. It
> > >>> > took about more than three hours. However when saving the same
> > >>> > HashMap in
> > a
> > >>> file
> > >>> > in the text format with the redirected System.out, it took only
> > >>> > 4.5
> > >>> seconds!
> > >>> >
> > >>> > Why is HBase so slow? It is indexing?
> > >>> >
> > >>> > My code to save data in HBase is as follows. I think the code
> > >>> > must be correct.
> > >>> >
> > >>> > ......
> > >>> > public synchronized void
> > >>> > AddVirtualOutgoingHHNeighbors(ConcurrentHashMap<String,
> > >>> > ConcurrentHashMap<String, Set<String>>> hhOutNeighborMap, int
-
Re: HBase Is So Slow To Save Data?Young Y Kim 2012-08-30, 08:44
In my experience , insert data under 15k/s per region server to avoid gc,
compaction.
On Thu, Aug 30, 2012 at 1:45 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear Cristofer,
>
> Thanks so much for your reminding!
>
> Best regards,
> Bing
>
> On Thu, Aug 30, 2012 at 12:32 AM, Cristofer Weber <
> [EMAIL PROTECTED]> wrote:
>
> > There's also a lot of conversions from same values to byte array
> > representation, eg, your NeighborStructure constants. You should do this
> > conversion only once to save time, since you are doing this inside 3
> nested
> > loops. Not sure about how much this can improve, but you should try this
> > also.
> >
> > Best regards,
> > Cristofer
> >
> > -----Mensagem original-----
> > De: Bing Li [mailto:[EMAIL PROTECTED]]
> > Enviada em: quarta-feira, 29 de agosto de 2012 13:07
> > Para: [EMAIL PROTECTED]
> > Cc: [EMAIL PROTECTED]
> > Assunto: Re: HBase Is So Slow To Save Data?
> >
> > I see. Thanks so much!
> >
> > Bing
> >
> >
> > On Wed, Aug 29, 2012 at 11:59 PM, N Keywal <[EMAIL PROTECTED]> wrote:
> >
> > > It's not useful here: if you have a memory issue, it's when your using
> > > the list, not when you have finished with it and set it to null.
> > > You need to monitor the memory consumption of the jvm, both the client
> > > & the server.
> > > Google around these keywords, there are many examples on the web.
> > > Google as well arrayList initialization.
> > >
> > > Note as well that the important is not the memory size of the
> > > structure on disk but the size of the" List<Put> puts = new
> > > ArrayList<Put>();" before the table put.
> > >
> > > On Wed, Aug 29, 2012 at 5:42 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > >
> > > > Dear N Keywal,
> > > >
> > > > Thanks so much for your reply!
> > > >
> > > > The total amount of data is about 110M. The available memory is
> > > > enough,
> > > 2G.
> > > >
> > > > In Java, I just set a collection to NULL to collect garbage. Do you
> > > > think it is fine?
> > > >
> > > > Best regards,
> > > > Bing
> > > >
> > > >
> > > > On Wed, Aug 29, 2012 at 11:22 PM, N Keywal <[EMAIL PROTECTED]>
> wrote:
> > > >
> > > >> Hi Bing,
> > > >>
> > > >> You should expect HBase to be slower in the generic case:
> > > >> 1) it writes much more data (see hbase data model), with extra
> > > >> columns qualifiers, timestamps & so on.
> > > >> 2) the data is written multiple times: once in the write-ahead-log,
> > > >> once per replica on datanode & so on again.
> > > >> 3) there are inter process calls & inter machine calls on the
> > > >> critical path.
> > > >>
> > > >> This is the cost of the atomicity, reliability and scalability
> > features.
> > > >> With these features in mind, HBase is reasonably fast to save data
> > > >> on a cluster.
> > > >>
> > > >> On your specific case (without the points 2 & 3 above), the
> > > >> performance seems to be very bad.
> > > >>
> > > >> You should first look at:
> > > >> - how much is spent in the put vs. preparing the list
> > > >> - do you have garbage collection going on? even swap?
> > > >> - what's the size of your final Array vs. the available memory?
> > > >>
> > > >> Cheers,
> > > >>
> > > >> N.
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > > >>
> > > >>> Dear all,
> > > >>>
> > > >>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
> > > >>>
> > > >>> Best regards,
> > > >>> Bing
> > > >>>
> > > >>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]>
> wrote:
> > > >>>
> > > >>> > Dear all,
> > > >>> >
> > > >>> > According to my experiences, it is very slow for HBase to save
> > data?
> > > >>> Am I
> > > >>> > right?
> > > >>> >
> > > >>> > For example, today I need to save data in a HashMap to HBase. It
> > > >>> > took about more than three hours. However when saving the same
> > > >>> > HashMap in
> > > a
> > > >>> file
> > > >>> > in the text format with the redirected System.out, it took only
> > > >>> > 4.5
compaction.
On Thu, Aug 30, 2012 at 1:45 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear Cristofer,
>
> Thanks so much for your reminding!
>
> Best regards,
> Bing
>
> On Thu, Aug 30, 2012 at 12:32 AM, Cristofer Weber <
> [EMAIL PROTECTED]> wrote:
>
> > There's also a lot of conversions from same values to byte array
> > representation, eg, your NeighborStructure constants. You should do this
> > conversion only once to save time, since you are doing this inside 3
> nested
> > loops. Not sure about how much this can improve, but you should try this
> > also.
> >
> > Best regards,
> > Cristofer
> >
> > -----Mensagem original-----
> > De: Bing Li [mailto:[EMAIL PROTECTED]]
> > Enviada em: quarta-feira, 29 de agosto de 2012 13:07
> > Para: [EMAIL PROTECTED]
> > Cc: [EMAIL PROTECTED]
> > Assunto: Re: HBase Is So Slow To Save Data?
> >
> > I see. Thanks so much!
> >
> > Bing
> >
> >
> > On Wed, Aug 29, 2012 at 11:59 PM, N Keywal <[EMAIL PROTECTED]> wrote:
> >
> > > It's not useful here: if you have a memory issue, it's when your using
> > > the list, not when you have finished with it and set it to null.
> > > You need to monitor the memory consumption of the jvm, both the client
> > > & the server.
> > > Google around these keywords, there are many examples on the web.
> > > Google as well arrayList initialization.
> > >
> > > Note as well that the important is not the memory size of the
> > > structure on disk but the size of the" List<Put> puts = new
> > > ArrayList<Put>();" before the table put.
> > >
> > > On Wed, Aug 29, 2012 at 5:42 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > >
> > > > Dear N Keywal,
> > > >
> > > > Thanks so much for your reply!
> > > >
> > > > The total amount of data is about 110M. The available memory is
> > > > enough,
> > > 2G.
> > > >
> > > > In Java, I just set a collection to NULL to collect garbage. Do you
> > > > think it is fine?
> > > >
> > > > Best regards,
> > > > Bing
> > > >
> > > >
> > > > On Wed, Aug 29, 2012 at 11:22 PM, N Keywal <[EMAIL PROTECTED]>
> wrote:
> > > >
> > > >> Hi Bing,
> > > >>
> > > >> You should expect HBase to be slower in the generic case:
> > > >> 1) it writes much more data (see hbase data model), with extra
> > > >> columns qualifiers, timestamps & so on.
> > > >> 2) the data is written multiple times: once in the write-ahead-log,
> > > >> once per replica on datanode & so on again.
> > > >> 3) there are inter process calls & inter machine calls on the
> > > >> critical path.
> > > >>
> > > >> This is the cost of the atomicity, reliability and scalability
> > features.
> > > >> With these features in mind, HBase is reasonably fast to save data
> > > >> on a cluster.
> > > >>
> > > >> On your specific case (without the points 2 & 3 above), the
> > > >> performance seems to be very bad.
> > > >>
> > > >> You should first look at:
> > > >> - how much is spent in the put vs. preparing the list
> > > >> - do you have garbage collection going on? even swap?
> > > >> - what's the size of your final Array vs. the available memory?
> > > >>
> > > >> Cheers,
> > > >>
> > > >> N.
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Aug 29, 2012 at 4:08 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > > >>
> > > >>> Dear all,
> > > >>>
> > > >>> By the way, my HBase is in the pseudo-distributed mode. Thanks!
> > > >>>
> > > >>> Best regards,
> > > >>> Bing
> > > >>>
> > > >>> On Wed, Aug 29, 2012 at 10:04 PM, Bing Li <[EMAIL PROTECTED]>
> wrote:
> > > >>>
> > > >>> > Dear all,
> > > >>> >
> > > >>> > According to my experiences, it is very slow for HBase to save
> > data?
> > > >>> Am I
> > > >>> > right?
> > > >>> >
> > > >>> > For example, today I need to save data in a HashMap to HBase. It
> > > >>> > took about more than three hours. However when saving the same
> > > >>> > HashMap in
> > > a
> > > >>> file
> > > >>> > in the text format with the redirected System.out, it took only
> > > >>> > 4.5
