Kumar, Deepak8
2013-03-12, 05:51
lars hofhansl
2013-03-12, 06:01
Kumar, Deepak8
2013-03-12, 06:27
Kumar, Deepak8
2013-03-12, 06:59
Kumar, Deepak8
2013-03-12, 11:46
Ted Yu
2013-03-12, 16:29
Himanshu Vashishtha
2013-03-12, 16:59
Gary Helmling
2013-03-12, 18:13
Kumar, Deepak8
2013-03-13, 15:19
Ted Yu
2013-03-13, 16:01
Himanshu Vashishtha
2013-03-13, 16:08
Kumar, Deepak8
2013-03-14, 17:09
Ted Yu
2013-03-14, 17:15
Himanshu Vashishtha
2013-03-14, 17:45
Anoop Sam John
2013-03-15, 06:55
Kumar, Deepak8
2013-03-20, 07:41
ramkrishna vasudevan
2013-03-20, 08:01
Anoop Sam John
2013-03-20, 08:36
Kumar, Deepak8
2013-03-20, 12:44
Anoop Sam John
2013-03-20, 12:58
Kumar, Deepak8
2013-03-20, 13:18
Kumar, Deepak8
2013-03-25, 16:53
Anoop Sam John
2013-03-26, 06:20
Kumar, Deepak8
2013-03-26, 07:27
Adrien Mogenet
2013-03-26, 07:42
Kumar, Deepak8
2013-03-26, 08:27
Anoop John
2013-03-26, 17:17
Kumar, Deepak8
2013-03-28, 10:50
ramkrishna vasudevan
2013-03-28, 10:53
Kumar, Deepak8
2013-03-28, 12:11
Agarwal, Saurabh
2013-03-28, 12:26
Anoop Sam John
2013-04-02, 06:51
-
Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-12, 05:51
Hi,
I have a table in hbase which has more than 5GB of data, it is distributed at 101 regions at 5 regionservers.
When I execute an endpoint which is supposed to fetch a column qualifier value using an endpoint RPC client, the region server goes down. The hbase master log says "Can't connect to region, retrying.." The same endpoint works fine for tables which has 300 records.
Could you please guide me the reason for being regionserver down?
Regards,
Deepak
I have a table in hbase which has more than 5GB of data, it is distributed at 101 regions at 5 regionservers.
When I execute an endpoint which is supposed to fetch a column qualifier value using an endpoint RPC client, the region server goes down. The hbase master log says "Can't connect to region, retrying.." The same endpoint works fine for tables which has 300 records.
Could you please guide me the reason for being regionserver down?
Regards,
Deepak
-
Re: Regionserver goes down while endpoint executionlars hofhansl 2013-03-12, 06:01
What does the region server log say?
Endpoints do not run in a sandbox. You can call System.exit(...) and your RegionServer will happily exit.
If you can, please show us your endpoint code.
-- Lars
________________________________
From: "Kumar, Deepak8 " <[EMAIL PROTECTED]>
To: "'[EMAIL PROTECTED]'" <[EMAIL PROTECTED]>
Sent: Monday, March 11, 2013 10:51 PM
Subject: Regionserver goes down while endpoint execution
Hi,
I have a table in hbase which has more than 5GB of data, it is distributed at 101 regions at 5 regionservers.
When I execute an endpoint which is supposed to fetch a column qualifier value using an endpoint RPC client, the region server goes down. The hbase master log says "Can't connect to region, retrying.." The same endpoint works fine for tables which has 300 records.
Could you please guide me the reason for being regionserver down?
Regards,
Deepak
Endpoints do not run in a sandbox. You can call System.exit(...) and your RegionServer will happily exit.
If you can, please show us your endpoint code.
-- Lars
________________________________
From: "Kumar, Deepak8 " <[EMAIL PROTECTED]>
To: "'[EMAIL PROTECTED]'" <[EMAIL PROTECTED]>
Sent: Monday, March 11, 2013 10:51 PM
Subject: Regionserver goes down while endpoint execution
Hi,
I have a table in hbase which has more than 5GB of data, it is distributed at 101 regions at 5 regionservers.
When I execute an endpoint which is supposed to fetch a column qualifier value using an endpoint RPC client, the region server goes down. The hbase master log says "Can't connect to region, retrying.." The same endpoint works fine for tables which has 300 records.
Could you please guide me the reason for being regionserver down?
Regards,
Deepak
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-12, 06:27
Lars,
Thanks for your quick response.There is not much info in region server log. I am again executing it with DEBUG log level in region servers.
Here is the endpoint code
public class ColumnAggregationEndpoint extends BaseEndpointCoprocessor
implements ColumnAggregationProtocol {
@Override
public List<String> getValues(byte[] family, byte[] qualifier, int batchSize, int cacheSize)
throws IOException {
// aggregate at each region
Scan scan = new Scan();
scan.addColumn(family, qualifier);
scan.setCaching(cacheSize);
scan.setBatch(batchSize);
List<String> values = new ArrayList<String>();
RegionCoprocessorEnvironment environment
(RegionCoprocessorEnvironment) getEnvironment();
InternalScanner scanner = environment.getRegion().getScanner(scan);
try {
List<KeyValue> curVals = new ArrayList<KeyValue>();
boolean hasMore = false;
do {
curVals.clear();
hasMore = scanner.next(curVals);
KeyValue kv = curVals.get(0);
values.add(Bytes.toString(kv.getValue()));
} while (hasMore);
} finally {
scanner.close();
}
return values;
}
}
The RPC client to invoke the Endpoint is as follows:
public class HBaseEndPointClientForElfLog {
public static void main(String[] args) {
try {
Configuration conf = HBaseConfiguration.create();
conf.set(
"hbase.zookeeper.quorum",
"vm-ab1f-dd21.nam.nsroot.net,vm-cb03-2277.nam.nsroot.net,vm-15c2-3bbf.nam.nsroot.net");
String tableName = "elf_log";
final String columnFamily = "content";
final String columnQualifier = "logFileName";
final String startRowKey = "153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net:";
final String endRowKey = "153299:1362953388000";
HTableInterface table = new HTable(conf, tableName);
Scan scan;
Map<byte[], List<String>> results;
// scan: for all regions
scan = new Scan();
results = table.coprocessorExec(ColumnAggregationProtocol.class,
startRowKey.getBytes(), endRowKey.getBytes(),
new Batch.Call<ColumnAggregationProtocol, List<String>>() {
public List<String> call(ColumnAggregationProtocol instance)
throws IOException {
return instance.getValues(columnFamily.getBytes(),
columnQualifier.getBytes(),2,5);
}
});
for (Map.Entry<byte[], List<String>> e : results.entrySet()) {
System.out.println("Size of list returned: "+e.getValue().size());
for(String singleVal: e.getValue()){
System.out.println(singleVal);
}
}
} catch (Throwable throwable) {
throwable.printStackTrace();
}
}
}
Regards,
Deepak
-----Original Message-----
From: lars hofhansl [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 12, 2013 2:01 AM
To: [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
What does the region server log say?
Endpoints do not run in a sandbox. You can call System.exit(...) and your RegionServer will happily exit.
If you can, please show us your endpoint code.
-- Lars
________________________________
From: "Kumar, Deepak8 " <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
To: "'[EMAIL PROTECTED]'" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Sent: Monday, March 11, 2013 10:51 PM
Subject: Regionserver goes down while endpoint execution
Hi,
I have a table in hbase which has more than 5GB of data, it is distributed at 101 regions at 5 regionservers.
When I execute an endpoint which is supposed to fetch a column qualifier value using an endpoint RPC client, the region server goes down. The hbase master log says "Can't connect to region, retrying.." The same endpoint works fine for tables which has 300 records.
Could you please guide me the reason for being regionserver down?
Regards,
Deepak
Thanks for your quick response.There is not much info in region server log. I am again executing it with DEBUG log level in region servers.
Here is the endpoint code
public class ColumnAggregationEndpoint extends BaseEndpointCoprocessor
implements ColumnAggregationProtocol {
@Override
public List<String> getValues(byte[] family, byte[] qualifier, int batchSize, int cacheSize)
throws IOException {
// aggregate at each region
Scan scan = new Scan();
scan.addColumn(family, qualifier);
scan.setCaching(cacheSize);
scan.setBatch(batchSize);
List<String> values = new ArrayList<String>();
RegionCoprocessorEnvironment environment
(RegionCoprocessorEnvironment) getEnvironment();
InternalScanner scanner = environment.getRegion().getScanner(scan);
try {
List<KeyValue> curVals = new ArrayList<KeyValue>();
boolean hasMore = false;
do {
curVals.clear();
hasMore = scanner.next(curVals);
KeyValue kv = curVals.get(0);
values.add(Bytes.toString(kv.getValue()));
} while (hasMore);
} finally {
scanner.close();
}
return values;
}
}
The RPC client to invoke the Endpoint is as follows:
public class HBaseEndPointClientForElfLog {
public static void main(String[] args) {
try {
Configuration conf = HBaseConfiguration.create();
conf.set(
"hbase.zookeeper.quorum",
"vm-ab1f-dd21.nam.nsroot.net,vm-cb03-2277.nam.nsroot.net,vm-15c2-3bbf.nam.nsroot.net");
String tableName = "elf_log";
final String columnFamily = "content";
final String columnQualifier = "logFileName";
final String startRowKey = "153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net:";
final String endRowKey = "153299:1362953388000";
HTableInterface table = new HTable(conf, tableName);
Scan scan;
Map<byte[], List<String>> results;
// scan: for all regions
scan = new Scan();
results = table.coprocessorExec(ColumnAggregationProtocol.class,
startRowKey.getBytes(), endRowKey.getBytes(),
new Batch.Call<ColumnAggregationProtocol, List<String>>() {
public List<String> call(ColumnAggregationProtocol instance)
throws IOException {
return instance.getValues(columnFamily.getBytes(),
columnQualifier.getBytes(),2,5);
}
});
for (Map.Entry<byte[], List<String>> e : results.entrySet()) {
System.out.println("Size of list returned: "+e.getValue().size());
for(String singleVal: e.getValue()){
System.out.println(singleVal);
}
}
} catch (Throwable throwable) {
throwable.printStackTrace();
}
}
}
Regards,
Deepak
-----Original Message-----
From: lars hofhansl [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 12, 2013 2:01 AM
To: [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
What does the region server log say?
Endpoints do not run in a sandbox. You can call System.exit(...) and your RegionServer will happily exit.
If you can, please show us your endpoint code.
-- Lars
________________________________
From: "Kumar, Deepak8 " <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
To: "'[EMAIL PROTECTED]'" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Sent: Monday, March 11, 2013 10:51 PM
Subject: Regionserver goes down while endpoint execution
Hi,
I have a table in hbase which has more than 5GB of data, it is distributed at 101 regions at 5 regionservers.
When I execute an endpoint which is supposed to fetch a column qualifier value using an endpoint RPC client, the region server goes down. The hbase master log says "Can't connect to region, retrying.." The same endpoint works fine for tables which has 300 records.
Could you please guide me the reason for being regionserver down?
Regards,
Deepak
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-12, 06:59
Lars,
It is having following errors when I execute the Endpoint RPC client from eclipse. It seems some of the regions at regionserver vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.
Could you guide how to fix it. I don't find any option to set hbase.rpc.timeout from hbase configuration menu in CDH4 CM server for hbase configuration.
Regards,
Deepak
3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid = 0x53d591b77090026, negotiated timeout = 60000
Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration warnOnceIfDeprecated
WARNING: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
Mar 12, 2013 2:44:00 AM org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation processExecs
WARNING: Error executing for row 153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net:
java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Tue Mar 12 02:34:15 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2271 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:35:16 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2403 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:36:18 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2465 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:37:20 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2500 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:38:22 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2538 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:39:25 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2572 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:40:30 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2606 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:41:34 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2640 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:42:43 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2677 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:44:00 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2842 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processExecs(HConnectionManager.java:1466)
at org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java:1577)
at org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java:1557)
at com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog.main(HBaseEndPointClientForElfLog.java:33)
Caused by: org.apa
It is having following errors when I execute the Endpoint RPC client from eclipse. It seems some of the regions at regionserver vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.
Could you guide how to fix it. I don't find any option to set hbase.rpc.timeout from hbase configuration menu in CDH4 CM server for hbase configuration.
Regards,
Deepak
3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid = 0x53d591b77090026, negotiated timeout = 60000
Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration warnOnceIfDeprecated
WARNING: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
Mar 12, 2013 2:44:00 AM org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation processExecs
WARNING: Error executing for row 153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net:
java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Tue Mar 12 02:34:15 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2271 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:35:16 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2403 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:36:18 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2465 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:37:20 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2500 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:38:22 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2538 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:39:25 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2572 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:40:30 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2606 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:41:34 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2640 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:42:43 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2677 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:44:00 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2842 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processExecs(HConnectionManager.java:1466)
at org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java:1577)
at org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java:1557)
at com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog.main(HBaseEndPointClientForElfLog.java:33)
Caused by: org.apa
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-12, 11:46
Lars,
I am getting following errors at datanode & region servers.
Regards,
Deepak
From: Kumar, Deepak8 [CCC-OT_IT NE]
Sent: Tuesday, March 12, 2013 3:00 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; 'lars hofhansl'
Subject: RE: Regionserver goes down while endpoint execution
Lars,
It is having following errors when I execute the Endpoint RPC client from eclipse. It seems some of the regions at regionserver vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.
Could you guide how to fix it. I don't find any option to set hbase.rpc.timeout from hbase configuration menu in CDH4 CM server for hbase configuration.
Regards,
Deepak
3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid = 0x53d591b77090026, negotiated timeout = 60000
Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration warnOnceIfDeprecated
WARNING: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
Mar 12, 2013 2:44:00 AM org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation processExecs
WARNING: Error executing for row 153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net:
java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Tue Mar 12 02:34:15 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2271 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:35:16 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2403 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:36:18 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2465 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:37:20 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2500 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:38:22 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2538 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:39:25 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2572 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:40:30 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2606 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:41:34 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2640 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:42:43 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2677 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:44:00 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.Soc
I am getting following errors at datanode & region servers.
Regards,
Deepak
From: Kumar, Deepak8 [CCC-OT_IT NE]
Sent: Tuesday, March 12, 2013 3:00 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; 'lars hofhansl'
Subject: RE: Regionserver goes down while endpoint execution
Lars,
It is having following errors when I execute the Endpoint RPC client from eclipse. It seems some of the regions at regionserver vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.
Could you guide how to fix it. I don't find any option to set hbase.rpc.timeout from hbase configuration menu in CDH4 CM server for hbase configuration.
Regards,
Deepak
3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid = 0x53d591b77090026, negotiated timeout = 60000
Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration warnOnceIfDeprecated
WARNING: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
Mar 12, 2013 2:44:00 AM org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation processExecs
WARNING: Error executing for row 153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net:
java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Tue Mar 12 02:34:15 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2271 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:35:16 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2403 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:36:18 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2465 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:37:20 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2500 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:38:22 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2538 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:39:25 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2572 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:40:30 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2606 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:41:34 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2640 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:42:43 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.SocketTimeoutException: Call to vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/150.110.96.212:2677 remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
Tue Mar 12 02:44:00 EDT 2013, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, java.net.Soc
-
Re: Regionserver goes down while endpoint executionTed Yu 2013-03-12, 16:29
>From region server log:
2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
making BlockReader. Closing stale
Socket[addr=/10.42.105.112,port=50010,localport=54114]
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
What version of HBase and hadoop are you using ?
Do versions of hadoop on Eclipse machine and in your cluster match ?
Cheers
On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Lars,****
>
> I am getting following errors at datanode & region servers.****
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> *Sent:* Tuesday, March 12, 2013 3:00 AM
> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; 'lars
> hofhansl'
>
> *Subject:* RE: Regionserver goes down while endpoint execution****
>
> ** **
>
> Lars,****
>
> It is having following errors when I execute the Endpoint RPC client from
> eclipse. It seems some of the regions at regionserver
> vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.****
>
> ** **
>
> Could you guide how to fix it. I don’t find any option to set hbase.rpc.timeout
> from hbase configuration menu in CDH4 CM server for hbase configuration.**
> **
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete
> on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid > 0x53d591b77090026, negotiated timeout = 60000****
>
> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
> warnOnceIfDeprecated****
>
> WARNING: hadoop.native.lib is deprecated. Instead, use
> io.native.lib.available****
>
> Mar 12, 2013 2:44:00 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> processExecs****
>
> WARNING: Error executing for row 153299:1362780381523:2932572079500658:
> vm-ab1f-dd21.nam.nsroot.net:****
>
> *java.util.concurrent.ExecutionException*: *
> org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed after
> attempts=10, exceptions:****
>
> Tue Mar 12 02:34:15 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> java.net.SocketTimeoutException*: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2271remote> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****
>
> Tue Mar 12 02:35:16 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> java.net.SocketTimeoutException*: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2403remote> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****
>
> Tue Mar 12 02:36:18 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> java.net.SocketTimeoutException*: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2465remote> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****
>
> Tue Mar 12 02:37:20 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> java.net.SocketTimeoutException*: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2500remote> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****
2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
making BlockReader. Closing stale
Socket[addr=/10.42.105.112,port=50010,localport=54114]
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
What version of HBase and hadoop are you using ?
Do versions of hadoop on Eclipse machine and in your cluster match ?
Cheers
On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Lars,****
>
> I am getting following errors at datanode & region servers.****
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> *Sent:* Tuesday, March 12, 2013 3:00 AM
> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; 'lars
> hofhansl'
>
> *Subject:* RE: Regionserver goes down while endpoint execution****
>
> ** **
>
> Lars,****
>
> It is having following errors when I execute the Endpoint RPC client from
> eclipse. It seems some of the regions at regionserver
> vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.****
>
> ** **
>
> Could you guide how to fix it. I don’t find any option to set hbase.rpc.timeout
> from hbase configuration menu in CDH4 CM server for hbase configuration.**
> **
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete
> on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid > 0x53d591b77090026, negotiated timeout = 60000****
>
> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
> warnOnceIfDeprecated****
>
> WARNING: hadoop.native.lib is deprecated. Instead, use
> io.native.lib.available****
>
> Mar 12, 2013 2:44:00 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> processExecs****
>
> WARNING: Error executing for row 153299:1362780381523:2932572079500658:
> vm-ab1f-dd21.nam.nsroot.net:****
>
> *java.util.concurrent.ExecutionException*: *
> org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed after
> attempts=10, exceptions:****
>
> Tue Mar 12 02:34:15 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> java.net.SocketTimeoutException*: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2271remote> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****
>
> Tue Mar 12 02:35:16 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> java.net.SocketTimeoutException*: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2403remote> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****
>
> Tue Mar 12 02:36:18 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> java.net.SocketTimeoutException*: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2465remote> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****
>
> Tue Mar 12 02:37:20 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> java.net.SocketTimeoutException*: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2500remote> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****
-
Re: Regionserver goes down while endpoint executionHimanshu Vashishtha 2013-03-12, 16:59
I don't see RS dying with this. It says that it is taking more time
than 60 sec (default timeout for clients), and therefore it stops
processing the coprocessor call as the client is disconnected.
Is your cluster okay? how many rows in the table? Normal scan works
good? Can you share more about your cluster details (nodes, tables,
regions, data size, etc)?
Thanks,
Himanshu
On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]> wrote:
> Lars,
>
> I am getting following errors at datanode & region servers.
>
>
>
> Regards,
>
> Deepak
>
>
>
> From: Kumar, Deepak8 [CCC-OT_IT NE]
> Sent: Tuesday, March 12, 2013 3:00 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; 'lars hofhansl'
>
>
> Subject: RE: Regionserver goes down while endpoint execution
>
>
>
> Lars,
>
> It is having following errors when I execute the Endpoint RPC client from
> eclipse. It seems some of the regions at regionserver
> vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.
>
>
>
> Could you guide how to fix it. I don’t find any option to set
> hbase.rpc.timeout from hbase configuration menu in CDH4 CM server for hbase
> configuration.
>
>
>
> Regards,
>
> Deepak
>
>
>
> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete
> on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid > 0x53d591b77090026, negotiated timeout = 60000
>
> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
> warnOnceIfDeprecated
>
> WARNING: hadoop.native.lib is deprecated. Instead, use
> io.native.lib.available
>
> Mar 12, 2013 2:44:00 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> processExecs
>
> WARNING: Error executing for row
> 153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net:
>
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=10, exceptions:
>
> Tue Mar 12 02:34:15 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f,
> java.net.SocketTimeoutException: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2271
> remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
>
> Tue Mar 12 02:35:16 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f,
> java.net.SocketTimeoutException: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2403
> remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
>
> Tue Mar 12 02:36:18 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f,
> java.net.SocketTimeoutException: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2465
> remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
>
> Tue Mar 12 02:37:20 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f,
> java.net.SocketTimeoutException: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2500
> remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
>
> Tue Mar 12 02:38:22 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f,
> java.net.SocketTimeoutException: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
than 60 sec (default timeout for clients), and therefore it stops
processing the coprocessor call as the client is disconnected.
Is your cluster okay? how many rows in the table? Normal scan works
good? Can you share more about your cluster details (nodes, tables,
regions, data size, etc)?
Thanks,
Himanshu
On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]> wrote:
> Lars,
>
> I am getting following errors at datanode & region servers.
>
>
>
> Regards,
>
> Deepak
>
>
>
> From: Kumar, Deepak8 [CCC-OT_IT NE]
> Sent: Tuesday, March 12, 2013 3:00 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; 'lars hofhansl'
>
>
> Subject: RE: Regionserver goes down while endpoint execution
>
>
>
> Lars,
>
> It is having following errors when I execute the Endpoint RPC client from
> eclipse. It seems some of the regions at regionserver
> vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.
>
>
>
> Could you guide how to fix it. I don’t find any option to set
> hbase.rpc.timeout from hbase configuration menu in CDH4 CM server for hbase
> configuration.
>
>
>
> Regards,
>
> Deepak
>
>
>
> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete
> on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid > 0x53d591b77090026, negotiated timeout = 60000
>
> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
> warnOnceIfDeprecated
>
> WARNING: hadoop.native.lib is deprecated. Instead, use
> io.native.lib.available
>
> Mar 12, 2013 2:44:00 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> processExecs
>
> WARNING: Error executing for row
> 153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net:
>
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=10, exceptions:
>
> Tue Mar 12 02:34:15 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f,
> java.net.SocketTimeoutException: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2271
> remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
>
> Tue Mar 12 02:35:16 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f,
> java.net.SocketTimeoutException: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2403
> remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
>
> Tue Mar 12 02:36:18 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f,
> java.net.SocketTimeoutException: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2465
> remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
>
> Tue Mar 12 02:37:20 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f,
> java.net.SocketTimeoutException: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/150.110.96.212:2500
> remote=vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]
>
> Tue Mar 12 02:38:22 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f,
> java.net.SocketTimeoutException: Call to
> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
-
Re: Regionserver goes down while endpoint executionGary Helmling 2013-03-12, 18:13
To expand on what Himanshu said, your endpoint is doing an unbounded scan
on the region, so with a region with a lot of rows it's taking more than 60
seconds to run to the region end, which is why the client side of the call
is timing out. In addition you're building up an in memory list of all the
values for that qualifier in that region, which could cause you to bump
into OOM issues, depending on how big your values are and how sparse the
given column qualifier is. If you trigger an OOMException, then the region
server would abort.
For this usage specifically, though -- scanning through a single column
qualifier for all rows -- you would be better off just doing a normal
client side scan, ie. HTable.getScanner(). Then you will avoid the client
timeout and potential server-side memory issues.
On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> From region server log:
>
> 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
> making BlockReader. Closing stale
> Socket[addr=/10.42.105.112,port=50010,localport=54114]
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
>
> What version of HBase and hadoop are you using ?
> Do versions of hadoop on Eclipse machine and in your cluster match ?
>
> Cheers
>
> On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Lars,****
> >
> > I am getting following errors at datanode & region servers.****
> >
> > ** **
> >
> > Regards,****
> >
> > Deepak****
> >
> > ** **
> >
> > *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> > *Sent:* Tuesday, March 12, 2013 3:00 AM
> > *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; 'lars
> > hofhansl'
> >
> > *Subject:* RE: Regionserver goes down while endpoint execution****
> >
> > ** **
> >
> > Lars,****
> >
> > It is having following errors when I execute the Endpoint RPC client from
> > eclipse. It seems some of the regions at regionserver
> > vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.****
> >
> > ** **
> >
> > Could you guide how to fix it. I don’t find any option to set
> hbase.rpc.timeout
> > from hbase configuration menu in CDH4 CM server for hbase
> configuration.**
> > **
> >
> > ** **
> >
> > Regards,****
> >
> > Deepak****
> >
> > ** **
> >
> > 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment
> complete
> > on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid > > 0x53d591b77090026, negotiated timeout = 60000****
> >
> > Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
> > warnOnceIfDeprecated****
> >
> > WARNING: hadoop.native.lib is deprecated. Instead, use
> > io.native.lib.available****
> >
> > Mar 12, 2013 2:44:00 AM
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> > processExecs****
> >
> > WARNING: Error executing for row 153299:1362780381523:2932572079500658:
> > vm-ab1f-dd21.nam.nsroot.net:****
> >
> > *java.util.concurrent.ExecutionException*: *
> > org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed after
> > attempts=10, exceptions:****
> >
> > Tue Mar 12 02:34:15 EDT 2013,
> > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> > java.net.SocketTimeoutException*: Call to
> > vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> > exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
> > waiting for channel to be ready for read. ch :
> > java.nio.channels.SocketChannel[connected local=/150.110.96.212:2271
> remote> > vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****
> >
> > Tue Mar 12 02:35:16 EDT 2013,
> > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> > java.net.SocketTimeoutException*: Call to
> > vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> > exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
on the region, so with a region with a lot of rows it's taking more than 60
seconds to run to the region end, which is why the client side of the call
is timing out. In addition you're building up an in memory list of all the
values for that qualifier in that region, which could cause you to bump
into OOM issues, depending on how big your values are and how sparse the
given column qualifier is. If you trigger an OOMException, then the region
server would abort.
For this usage specifically, though -- scanning through a single column
qualifier for all rows -- you would be better off just doing a normal
client side scan, ie. HTable.getScanner(). Then you will avoid the client
timeout and potential server-side memory issues.
On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> From region server log:
>
> 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
> making BlockReader. Closing stale
> Socket[addr=/10.42.105.112,port=50010,localport=54114]
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
>
> What version of HBase and hadoop are you using ?
> Do versions of hadoop on Eclipse machine and in your cluster match ?
>
> Cheers
>
> On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Lars,****
> >
> > I am getting following errors at datanode & region servers.****
> >
> > ** **
> >
> > Regards,****
> >
> > Deepak****
> >
> > ** **
> >
> > *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> > *Sent:* Tuesday, March 12, 2013 3:00 AM
> > *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; 'lars
> > hofhansl'
> >
> > *Subject:* RE: Regionserver goes down while endpoint execution****
> >
> > ** **
> >
> > Lars,****
> >
> > It is having following errors when I execute the Endpoint RPC client from
> > eclipse. It seems some of the regions at regionserver
> > vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.****
> >
> > ** **
> >
> > Could you guide how to fix it. I don’t find any option to set
> hbase.rpc.timeout
> > from hbase configuration menu in CDH4 CM server for hbase
> configuration.**
> > **
> >
> > ** **
> >
> > Regards,****
> >
> > Deepak****
> >
> > ** **
> >
> > 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment
> complete
> > on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid > > 0x53d591b77090026, negotiated timeout = 60000****
> >
> > Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
> > warnOnceIfDeprecated****
> >
> > WARNING: hadoop.native.lib is deprecated. Instead, use
> > io.native.lib.available****
> >
> > Mar 12, 2013 2:44:00 AM
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> > processExecs****
> >
> > WARNING: Error executing for row 153299:1362780381523:2932572079500658:
> > vm-ab1f-dd21.nam.nsroot.net:****
> >
> > *java.util.concurrent.ExecutionException*: *
> > org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed after
> > attempts=10, exceptions:****
> >
> > Tue Mar 12 02:34:15 EDT 2013,
> > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> > java.net.SocketTimeoutException*: Call to
> > vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> > exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
> > waiting for channel to be ready for read. ch :
> > java.nio.channels.SocketChannel[connected local=/150.110.96.212:2271
> remote> > vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****
> >
> > Tue Mar 12 02:35:16 EDT 2013,
> > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f, *
> > java.net.SocketTimeoutException*: Call to
> > vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020 failed on socket timeout
> > exception: *java.net.SocketTimeoutException*: 60000 millis timeout while
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-13, 15:19
Thanks guys for assisting. I am getting OOM exception yet. I have one query about Endpoints. As endpoint executes in parallel, so if I have a table which is distributed at 101 regions across 5 regionserver. Would it be 101 threads of endpoint executing in parallel?
Regards,
Deepak
From: Gary Helmling [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 12, 2013 2:14 PM
To: [EMAIL PROTECTED]
Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]
Subject: Re: Regionserver goes down while endpoint execution
To expand on what Himanshu said, your endpoint is doing an unbounded scan on the region, so with a region with a lot of rows it's taking more than 60 seconds to run to the region end, which is why the client side of the call is timing out. In addition you're building up an in memory list of all the values for that qualifier in that region, which could cause you to bump into OOM issues, depending on how big your values are and how sparse the given column qualifier is. If you trigger an OOMException, then the region server would abort.
For this usage specifically, though -- scanning through a single column qualifier for all rows -- you would be better off just doing a normal client side scan, ie. HTable.getScanner(). Then you will avoid the client timeout and potential server-side memory issues.
On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>From region server log:
2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
making BlockReader. Closing stale
Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localport=54114]
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
What version of HBase and hadoop are you using ?
Do versions of hadoop on Eclipse machine and in your cluster match ?
Cheers
On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Lars,****
>
> I am getting following errors at datanode & region servers.****
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> *Sent:* Tuesday, March 12, 2013 3:00 AM
> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>'; 'lars
> hofhansl'
>
> *Subject:* RE: Regionserver goes down while endpoint execution****
>
> ** **
>
> Lars,****
>
> It is having following errors when I execute the Endpoint RPC client from
> eclipse. It seems some of the regions at regionserver
> vm-8aa9-fe74.nam.nsroot.net<http://vm-8aa9-fe74.nam.nsroot.net> is taking more time to reponse.****
>
> ** **
>
> Could you guide how to fix it. I don't find any option to set hbase.rpc.timeout
> from hbase configuration menu in CDH4 CM server for hbase configuration.**
> **
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete
> on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181<http://vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181>, sessionid > 0x53d591b77090026, negotiated timeout = 60000****
>
> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
> warnOnceIfDeprecated****
>
> WARNING: hadoop.native.lib is deprecated. Instead, use
> io.native.lib.available****
>
> Mar 12, 2013 2:44:00 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> processExecs****
>
> WARNING: Error executing for row 153299:1362780381523:2932572079500658:
> vm-ab1f-dd21.nam.nsroot.net:****
>
> *java.util.concurrent.ExecutionException*: *
> org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed after
> attempts=10, exceptions:****
>
> Tue Mar 12 02:34:15 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, *
> java.net.SocketTimeoutException*: Call to
Regards,
Deepak
From: Gary Helmling [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 12, 2013 2:14 PM
To: [EMAIL PROTECTED]
Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]
Subject: Re: Regionserver goes down while endpoint execution
To expand on what Himanshu said, your endpoint is doing an unbounded scan on the region, so with a region with a lot of rows it's taking more than 60 seconds to run to the region end, which is why the client side of the call is timing out. In addition you're building up an in memory list of all the values for that qualifier in that region, which could cause you to bump into OOM issues, depending on how big your values are and how sparse the given column qualifier is. If you trigger an OOMException, then the region server would abort.
For this usage specifically, though -- scanning through a single column qualifier for all rows -- you would be better off just doing a normal client side scan, ie. HTable.getScanner(). Then you will avoid the client timeout and potential server-side memory issues.
On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>From region server log:
2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
making BlockReader. Closing stale
Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localport=54114]
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
What version of HBase and hadoop are you using ?
Do versions of hadoop on Eclipse machine and in your cluster match ?
Cheers
On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Lars,****
>
> I am getting following errors at datanode & region servers.****
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> *Sent:* Tuesday, March 12, 2013 3:00 AM
> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>'; 'lars
> hofhansl'
>
> *Subject:* RE: Regionserver goes down while endpoint execution****
>
> ** **
>
> Lars,****
>
> It is having following errors when I execute the Endpoint RPC client from
> eclipse. It seems some of the regions at regionserver
> vm-8aa9-fe74.nam.nsroot.net<http://vm-8aa9-fe74.nam.nsroot.net> is taking more time to reponse.****
>
> ** **
>
> Could you guide how to fix it. I don't find any option to set hbase.rpc.timeout
> from hbase configuration menu in CDH4 CM server for hbase configuration.**
> **
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete
> on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181<http://vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181>, sessionid > 0x53d591b77090026, negotiated timeout = 60000****
>
> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
> warnOnceIfDeprecated****
>
> WARNING: hadoop.native.lib is deprecated. Instead, use
> io.native.lib.available****
>
> Mar 12, 2013 2:44:00 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> processExecs****
>
> WARNING: Error executing for row 153299:1362780381523:2932572079500658:
> vm-ab1f-dd21.nam.nsroot.net:****
>
> *java.util.concurrent.ExecutionException*: *
> org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed after
> attempts=10, exceptions:****
>
> Tue Mar 12 02:34:15 EDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, *
> java.net.SocketTimeoutException*: Call to
-
Re: Regionserver goes down while endpoint executionTed Yu 2013-03-13, 16:01
bq. I am getting OOM exception yet.
Can you confirm whether you got OOME or not ?
bq. Would it be 101 threads of endpoint executing in parallel?
This signifies the need for your endpoint to use memory conservatively.
Cheers
On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Thanks guys for assisting. I am getting OOM exception yet. I have one
> query about Endpoints. As endpoint executes in parallel, so if I have a
> table which is distributed at 101 regions across 5 regionserver. Would it
> be 101 threads of endpoint executing in parallel?****
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> *From:* Gary Helmling [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, March 12, 2013 2:14 PM
> *To:* [EMAIL PROTECTED]
> *Cc:* lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]
>
> *Subject:* Re: Regionserver goes down while endpoint execution****
>
> ** **
>
> To expand on what Himanshu said, your endpoint is doing an unbounded scan
> on the region, so with a region with a lot of rows it's taking more than 60
> seconds to run to the region end, which is why the client side of the call
> is timing out. In addition you're building up an in memory list of all the
> values for that qualifier in that region, which could cause you to bump
> into OOM issues, depending on how big your values are and how sparse the
> given column qualifier is. If you trigger an OOMException, then the region
> server would abort.****
>
> ** **
>
> For this usage specifically, though -- scanning through a single column
> qualifier for all rows -- you would be better off just doing a normal
> client side scan, ie. HTable.getScanner(). Then you will avoid the client
> timeout and potential server-side memory issues.****
>
> ** **
>
> On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]> wrote:****
>
> From region server log:
>
> 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
> making BlockReader. Closing stale
> Socket[addr=/10.42.105.112,port=50010,localport=54114]
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
>
> What version of HBase and hadoop are you using ?
> Do versions of hadoop on Eclipse machine and in your cluster match ?
>
> Cheers****
>
>
> On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:****
>
> > Lars,****
> >
> > I am getting following errors at datanode & region servers.****
> >
> > ** **
> >
> > Regards,****
> >
> > Deepak****
> >
> > ** **
> >
> > *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> > *Sent:* Tuesday, March 12, 2013 3:00 AM
> > *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; 'lars
> > hofhansl'
> >
> > *Subject:* RE: Regionserver goes down while endpoint execution****
> >
> > ** **
> >
> > Lars,********
>
> >
> > It is having following errors when I execute the Endpoint RPC client from
> > eclipse. It seems some of the regions at regionserver****
>
> > vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.****
> >
> > ** ******
>
> >
> > Could you guide how to fix it. I don’t find any option to set
> hbase.rpc.timeout****
>
> > from hbase configuration menu in CDH4 CM server for hbase
> configuration.**
> > **
> >
> > ** **
> >
> > Regards,****
> >
> > Deepak****
> >
> > ** ******
>
> >
> > 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment
> complete
> > on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid =****
>
> > 0x53d591b77090026, negotiated timeout = 60000********
>
> >
> > Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration****
>
> > warnOnceIfDeprecated********
>
> >
> > WARNING: hadoop.native.lib is deprecated. Instead, use****
>
> > io.native.lib.available********
>
> >
> > Mar 12, 2013 2:44:00 AM
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
Can you confirm whether you got OOME or not ?
bq. Would it be 101 threads of endpoint executing in parallel?
This signifies the need for your endpoint to use memory conservatively.
Cheers
On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Thanks guys for assisting. I am getting OOM exception yet. I have one
> query about Endpoints. As endpoint executes in parallel, so if I have a
> table which is distributed at 101 regions across 5 regionserver. Would it
> be 101 threads of endpoint executing in parallel?****
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> *From:* Gary Helmling [mailto:[EMAIL PROTECTED]]
> *Sent:* Tuesday, March 12, 2013 2:14 PM
> *To:* [EMAIL PROTECTED]
> *Cc:* lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]
>
> *Subject:* Re: Regionserver goes down while endpoint execution****
>
> ** **
>
> To expand on what Himanshu said, your endpoint is doing an unbounded scan
> on the region, so with a region with a lot of rows it's taking more than 60
> seconds to run to the region end, which is why the client side of the call
> is timing out. In addition you're building up an in memory list of all the
> values for that qualifier in that region, which could cause you to bump
> into OOM issues, depending on how big your values are and how sparse the
> given column qualifier is. If you trigger an OOMException, then the region
> server would abort.****
>
> ** **
>
> For this usage specifically, though -- scanning through a single column
> qualifier for all rows -- you would be better off just doing a normal
> client side scan, ie. HTable.getScanner(). Then you will avoid the client
> timeout and potential server-side memory issues.****
>
> ** **
>
> On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]> wrote:****
>
> From region server log:
>
> 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
> making BlockReader. Closing stale
> Socket[addr=/10.42.105.112,port=50010,localport=54114]
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
>
> What version of HBase and hadoop are you using ?
> Do versions of hadoop on Eclipse machine and in your cluster match ?
>
> Cheers****
>
>
> On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:****
>
> > Lars,****
> >
> > I am getting following errors at datanode & region servers.****
> >
> > ** **
> >
> > Regards,****
> >
> > Deepak****
> >
> > ** **
> >
> > *From:* Kumar, Deepak8 [CCC-OT_IT NE]
> > *Sent:* Tuesday, March 12, 2013 3:00 AM
> > *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; 'lars
> > hofhansl'
> >
> > *Subject:* RE: Regionserver goes down while endpoint execution****
> >
> > ** **
> >
> > Lars,********
>
> >
> > It is having following errors when I execute the Endpoint RPC client from
> > eclipse. It seems some of the regions at regionserver****
>
> > vm-8aa9-fe74.nam.nsroot.net is taking more time to reponse.****
> >
> > ** ******
>
> >
> > Could you guide how to fix it. I don’t find any option to set
> hbase.rpc.timeout****
>
> > from hbase configuration menu in CDH4 CM server for hbase
> configuration.**
> > **
> >
> > ** **
> >
> > Regards,****
> >
> > Deepak****
> >
> > ** ******
>
> >
> > 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment
> complete
> > on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181, sessionid =****
>
> > 0x53d591b77090026, negotiated timeout = 60000********
>
> >
> > Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration****
>
> > warnOnceIfDeprecated********
>
> >
> > WARNING: hadoop.native.lib is deprecated. Instead, use****
>
> > io.native.lib.available********
>
> >
> > Mar 12, 2013 2:44:00 AM
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
-
Re: Regionserver goes down while endpoint executionHimanshu Vashishtha 2013-03-13, 16:08
On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]> wrote:
> Thanks guys for assisting. I am getting OOM exception yet. I have one query about Endpoints. As endpoint executes in parallel, so if I have a table which is distributed at 101 regions across 5 regionserver. Would it be 101 threads of endpoint executing in parallel?
No and Yes.
The endpoints are not processed as separate threads, they are
processed as just another request (via regionserver handlers). Yes,
the execution will be in parallel in the sense that a separate client
side call will be used for each of the regions that are in the range
you specify.
>
> Regards,
> Deepak
>
> From: Gary Helmling [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 12, 2013 2:14 PM
> To: [EMAIL PROTECTED]
> Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]
> Subject: Re: Regionserver goes down while endpoint execution
>
> To expand on what Himanshu said, your endpoint is doing an unbounded scan on the region, so with a region with a lot of rows it's taking more than 60 seconds to run to the region end, which is why the client side of the call is timing out. In addition you're building up an in memory list of all the values for that qualifier in that region, which could cause you to bump into OOM issues, depending on how big your values are and how sparse the given column qualifier is. If you trigger an OOMException, then the region server would abort.
>
> For this usage specifically, though -- scanning through a single column qualifier for all rows -- you would be better off just doing a normal client side scan, ie. HTable.getScanner(). Then you will avoid the client timeout and potential server-side memory issues.
>
> On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> From region server log:
>
> 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
> making BlockReader. Closing stale
> Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localport=54114]
> java.io.EOFException: Premature EOF: no length prefix available
> at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
>
> What version of HBase and hadoop are you using ?
> Do versions of hadoop on Eclipse machine and in your cluster match ?
>
> Cheers
>
> On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
>> Lars,****
>>
>> I am getting following errors at datanode & region servers.****
>>
>> ** **
>>
>> Regards,****
>>
>> Deepak****
>>
>> ** **
>>
>> *From:* Kumar, Deepak8 [CCC-OT_IT NE]
>> *Sent:* Tuesday, March 12, 2013 3:00 AM
>> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>'; 'lars
>> hofhansl'
>>
>> *Subject:* RE: Regionserver goes down while endpoint execution****
>>
>> ** **
>>
>> Lars,****
>>
>> It is having following errors when I execute the Endpoint RPC client from
>> eclipse. It seems some of the regions at regionserver
>> vm-8aa9-fe74.nam.nsroot.net<http://vm-8aa9-fe74.nam.nsroot.net> is taking more time to reponse.****
>>
>> ** **
>>
>> Could you guide how to fix it. I don't find any option to set hbase.rpc.timeout
>> from hbase configuration menu in CDH4 CM server for hbase configuration.**
>> **
>>
>> ** **
>>
>> Regards,****
>>
>> Deepak****
>>
>> ** **
>>
>> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete
>> on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181<http://vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181>, sessionid >> 0x53d591b77090026, negotiated timeout = 60000****
>>
>> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
>> warnOnceIfDeprecated****
>>
>> WARNING: hadoop.native.lib is deprecated. Instead, use
>> io.native.lib.available****
>>
>> Mar 12, 2013 2:44:00 AM
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> Thanks guys for assisting. I am getting OOM exception yet. I have one query about Endpoints. As endpoint executes in parallel, so if I have a table which is distributed at 101 regions across 5 regionserver. Would it be 101 threads of endpoint executing in parallel?
No and Yes.
The endpoints are not processed as separate threads, they are
processed as just another request (via regionserver handlers). Yes,
the execution will be in parallel in the sense that a separate client
side call will be used for each of the regions that are in the range
you specify.
>
> Regards,
> Deepak
>
> From: Gary Helmling [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 12, 2013 2:14 PM
> To: [EMAIL PROTECTED]
> Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]
> Subject: Re: Regionserver goes down while endpoint execution
>
> To expand on what Himanshu said, your endpoint is doing an unbounded scan on the region, so with a region with a lot of rows it's taking more than 60 seconds to run to the region end, which is why the client side of the call is timing out. In addition you're building up an in memory list of all the values for that qualifier in that region, which could cause you to bump into OOM issues, depending on how big your values are and how sparse the given column qualifier is. If you trigger an OOMException, then the region server would abort.
>
> For this usage specifically, though -- scanning through a single column qualifier for all rows -- you would be better off just doing a normal client side scan, ie. HTable.getScanner(). Then you will avoid the client timeout and potential server-side memory issues.
>
> On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> From region server log:
>
> 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
> making BlockReader. Closing stale
> Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localport=54114]
> java.io.EOFException: Premature EOF: no length prefix available
> at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:407)
>
> What version of HBase and hadoop are you using ?
> Do versions of hadoop on Eclipse machine and in your cluster match ?
>
> Cheers
>
> On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
>> Lars,****
>>
>> I am getting following errors at datanode & region servers.****
>>
>> ** **
>>
>> Regards,****
>>
>> Deepak****
>>
>> ** **
>>
>> *From:* Kumar, Deepak8 [CCC-OT_IT NE]
>> *Sent:* Tuesday, March 12, 2013 3:00 AM
>> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>'; 'lars
>> hofhansl'
>>
>> *Subject:* RE: Regionserver goes down while endpoint execution****
>>
>> ** **
>>
>> Lars,****
>>
>> It is having following errors when I execute the Endpoint RPC client from
>> eclipse. It seems some of the regions at regionserver
>> vm-8aa9-fe74.nam.nsroot.net<http://vm-8aa9-fe74.nam.nsroot.net> is taking more time to reponse.****
>>
>> ** **
>>
>> Could you guide how to fix it. I don't find any option to set hbase.rpc.timeout
>> from hbase configuration menu in CDH4 CM server for hbase configuration.**
>> **
>>
>> ** **
>>
>> Regards,****
>>
>> Deepak****
>>
>> ** **
>>
>> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment complete
>> on server vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181<http://vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181>, sessionid >> 0x53d591b77090026, negotiated timeout = 60000****
>>
>> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration
>> warnOnceIfDeprecated****
>>
>> WARNING: hadoop.native.lib is deprecated. Instead, use
>> io.native.lib.available****
>>
>> Mar 12, 2013 2:44:00 AM
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-14, 17:09
Hi,
It seems due to huge data the RegionServer is getting down. Now I am trying to fetch the data in parts & is running fine. I need some more info about the Endpoint execution:
My use case is to fetch the data from HBase as per some rowkey range & to render it at UI. Since endpoints are executed in parallel so I am looking to use it.
Ø Suppose I provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC client & these rowkeys are distributed at 5 regions across 4 region servers. If I fetch 10 records at a time, do we have any way to guarantee that it would come in serial order like first result would of rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 & the fetch would be from rowkey11 to rowkey20, irrespective of the region & region servers?
Regards,
Deepak
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Himanshu Vashishtha
Sent: Wednesday, March 13, 2013 12:09 PM
To: [EMAIL PROTECTED]
Cc: Gary Helmling; [EMAIL PROTECTED]; lars hofhansl
Subject: Re: Regionserver goes down while endpoint execution
On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> Thanks guys for assisting. I am getting OOM exception yet. I have one query about Endpoints. As endpoint executes in parallel, so if I have a table which is distributed at 101 regions across 5 regionserver. Would it be 101 threads of endpoint executing in parallel?
No and Yes.
The endpoints are not processed as separate threads, they are processed as just another request (via regionserver handlers). Yes, the execution will be in parallel in the sense that a separate client side call will be used for each of the regions that are in the range you specify.
>
> Regards,
> Deepak
>
> From: Gary Helmling [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 12, 2013 2:14 PM
> To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
> Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]
> Subject: Re: Regionserver goes down while endpoint execution
>
> To expand on what Himanshu said, your endpoint is doing an unbounded scan on the region, so with a region with a lot of rows it's taking more than 60 seconds to run to the region end, which is why the client side of the call is timing out. In addition you're building up an in memory list of all the values for that qualifier in that region, which could cause you to bump into OOM issues, depending on how big your values are and how sparse the given column qualifier is. If you trigger an OOMException, then the region server would abort.
>
> For this usage specifically, though -- scanning through a single column qualifier for all rows -- you would be better off just doing a normal client side scan, ie. HTable.getScanner(). Then you will avoid the client timeout and potential server-side memory issues.
>
> On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]%3cmailto:[EMAIL PROTECTED]>>> wrote:
> From region server log:
>
> 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
> making BlockReader. Closing stale
> Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localport
> 54114]
> java.io.EOFException: Premature EOF: no length prefix available
> at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockRe
> ader2.java:407)
>
> What version of HBase and hadoop are you using ?
> Do versions of hadoop on Eclipse machine and in your cluster match ?
>
> Cheers
>
> On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]%3cmailto:[EMAIL PROTECTED]>>>wrote:
>> Lars,****
>>
>> I am getting following errors at datanode & region servers.****
>>
>> ** **
>>
>> Regards,****
It seems due to huge data the RegionServer is getting down. Now I am trying to fetch the data in parts & is running fine. I need some more info about the Endpoint execution:
My use case is to fetch the data from HBase as per some rowkey range & to render it at UI. Since endpoints are executed in parallel so I am looking to use it.
Ø Suppose I provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC client & these rowkeys are distributed at 5 regions across 4 region servers. If I fetch 10 records at a time, do we have any way to guarantee that it would come in serial order like first result would of rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 & the fetch would be from rowkey11 to rowkey20, irrespective of the region & region servers?
Regards,
Deepak
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Himanshu Vashishtha
Sent: Wednesday, March 13, 2013 12:09 PM
To: [EMAIL PROTECTED]
Cc: Gary Helmling; [EMAIL PROTECTED]; lars hofhansl
Subject: Re: Regionserver goes down while endpoint execution
On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> Thanks guys for assisting. I am getting OOM exception yet. I have one query about Endpoints. As endpoint executes in parallel, so if I have a table which is distributed at 101 regions across 5 regionserver. Would it be 101 threads of endpoint executing in parallel?
No and Yes.
The endpoints are not processed as separate threads, they are processed as just another request (via regionserver handlers). Yes, the execution will be in parallel in the sense that a separate client side call will be used for each of the regions that are in the range you specify.
>
> Regards,
> Deepak
>
> From: Gary Helmling [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 12, 2013 2:14 PM
> To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
> Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]
> Subject: Re: Regionserver goes down while endpoint execution
>
> To expand on what Himanshu said, your endpoint is doing an unbounded scan on the region, so with a region with a lot of rows it's taking more than 60 seconds to run to the region end, which is why the client side of the call is timing out. In addition you're building up an in memory list of all the values for that qualifier in that region, which could cause you to bump into OOM issues, depending on how big your values are and how sparse the given column qualifier is. If you trigger an OOMException, then the region server would abort.
>
> For this usage specifically, though -- scanning through a single column qualifier for all rows -- you would be better off just doing a normal client side scan, ie. HTable.getScanner(). Then you will avoid the client timeout and potential server-side memory issues.
>
> On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]%3cmailto:[EMAIL PROTECTED]>>> wrote:
> From region server log:
>
> 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error
> making BlockReader. Closing stale
> Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localport
> 54114]
> java.io.EOFException: Premature EOF: no length prefix available
> at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockRe
> ader2.java:407)
>
> What version of HBase and hadoop are you using ?
> Do versions of hadoop on Eclipse machine and in your cluster match ?
>
> Cheers
>
> On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]%3cmailto:[EMAIL PROTECTED]>>>wrote:
>> Lars,****
>>
>> I am getting following errors at datanode & region servers.****
>>
>> ** **
>>
>> Regards,****
-
Re: Regionserver goes down while endpoint executionTed Yu 2013-03-14, 17:15
bq. provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC client
If I understand you correctly, you perform batching at the client as
described above.
The order would be as you expected.
Cheers
On Thu, Mar 14, 2013 at 10:09 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi,****
>
> It seems due to huge data the RegionServer is getting down. Now I am
> trying to fetch the data in parts & is running fine. I need some more info
> about the Endpoint execution:****
>
> ** **
>
> My use case is to fetch the data from HBase as per some rowkey range & to
> render it at UI. Since endpoints are executed in parallel so I am looking
> to use it. ****
>
> ** **
>
> **Ø **Suppose I provide the rowkey range as rowkey1 to rowkey100 in
> endpoint RPC client & these rowkeys are distributed at 5 regions across 4
> region servers. If I fetch 10 records at a time, do we have any way to
> guarantee that it would come in serial order like first result would of
> rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 & the
> fetch would be from rowkey11 to rowkey20, irrespective of the region &
> region servers?****
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> ** **
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of
> Himanshu Vashishtha
> Sent: Wednesday, March 13, 2013 12:09 PM
> To: [EMAIL PROTECTED]
> Cc: Gary Helmling; [EMAIL PROTECTED]; lars hofhansl
> Subject: Re: Regionserver goes down while endpoint execution
>
> ** **
>
> On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>
> wrote:****
>
> > Thanks guys for assisting. I am getting OOM exception yet. I have one
> query about Endpoints. As endpoint executes in parallel, so if I have a
> table which is distributed at 101 regions across 5 regionserver. Would it
> be 101 threads of endpoint executing in parallel?****
>
> ** **
>
> No and Yes.****
>
> ** **
>
> The endpoints are not processed as separate threads, they are processed as
> just another request (via regionserver handlers). Yes, the execution will
> be in parallel in the sense that a separate client side call will be used
> for each of the regions that are in the range you specify.****
>
> ** **
>
> >** **
>
> > Regards,****
>
> > Deepak****
>
> >** **
>
> > From: Gary Helmling [mailto:[EMAIL PROTECTED] <[EMAIL PROTECTED]>]**
> **
>
> > Sent: Tuesday, March 12, 2013 2:14 PM****
>
> > To: [EMAIL PROTECTED]****
>
> > Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]****
>
> > Subject: Re: Regionserver goes down while endpoint execution****
>
> >** **
>
> > To expand on what Himanshu said, your endpoint is doing an unbounded
> scan on the region, so with a region with a lot of rows it's taking more
> than 60 seconds to run to the region end, which is why the client side of
> the call is timing out. In addition you're building up an in memory list
> of all the values for that qualifier in that region, which could cause you
> to bump into OOM issues, depending on how big your values are and how
> sparse the given column qualifier is. If you trigger an OOMException, then
> the region server would abort.****
>
> >** **
>
> > For this usage specifically, though -- scanning through a single column
> qualifier for all rows -- you would be better off just doing a normal
> client side scan, ie. HTable.getScanner(). Then you will avoid the client
> timeout and potential server-side memory issues.****
>
> >** **
>
> > On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:****
>
> > From region server log:****
>
> >** **
>
> > 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error **
> **
>
> > making BlockReader. Closing stale ****
>
> > Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localport=**
> **
>
> > 54114]****
>
> > java.io.EOFException: Premature EOF: no length prefix available****
>
> > at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
If I understand you correctly, you perform batching at the client as
described above.
The order would be as you expected.
Cheers
On Thu, Mar 14, 2013 at 10:09 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi,****
>
> It seems due to huge data the RegionServer is getting down. Now I am
> trying to fetch the data in parts & is running fine. I need some more info
> about the Endpoint execution:****
>
> ** **
>
> My use case is to fetch the data from HBase as per some rowkey range & to
> render it at UI. Since endpoints are executed in parallel so I am looking
> to use it. ****
>
> ** **
>
> **Ø **Suppose I provide the rowkey range as rowkey1 to rowkey100 in
> endpoint RPC client & these rowkeys are distributed at 5 regions across 4
> region servers. If I fetch 10 records at a time, do we have any way to
> guarantee that it would come in serial order like first result would of
> rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 & the
> fetch would be from rowkey11 to rowkey20, irrespective of the region &
> region servers?****
>
> ** **
>
> Regards,****
>
> Deepak****
>
> ** **
>
> ** **
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of
> Himanshu Vashishtha
> Sent: Wednesday, March 13, 2013 12:09 PM
> To: [EMAIL PROTECTED]
> Cc: Gary Helmling; [EMAIL PROTECTED]; lars hofhansl
> Subject: Re: Regionserver goes down while endpoint execution
>
> ** **
>
> On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>
> wrote:****
>
> > Thanks guys for assisting. I am getting OOM exception yet. I have one
> query about Endpoints. As endpoint executes in parallel, so if I have a
> table which is distributed at 101 regions across 5 regionserver. Would it
> be 101 threads of endpoint executing in parallel?****
>
> ** **
>
> No and Yes.****
>
> ** **
>
> The endpoints are not processed as separate threads, they are processed as
> just another request (via regionserver handlers). Yes, the execution will
> be in parallel in the sense that a separate client side call will be used
> for each of the regions that are in the range you specify.****
>
> ** **
>
> >** **
>
> > Regards,****
>
> > Deepak****
>
> >** **
>
> > From: Gary Helmling [mailto:[EMAIL PROTECTED] <[EMAIL PROTECTED]>]**
> **
>
> > Sent: Tuesday, March 12, 2013 2:14 PM****
>
> > To: [EMAIL PROTECTED]****
>
> > Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]****
>
> > Subject: Re: Regionserver goes down while endpoint execution****
>
> >** **
>
> > To expand on what Himanshu said, your endpoint is doing an unbounded
> scan on the region, so with a region with a lot of rows it's taking more
> than 60 seconds to run to the region end, which is why the client side of
> the call is timing out. In addition you're building up an in memory list
> of all the values for that qualifier in that region, which could cause you
> to bump into OOM issues, depending on how big your values are and how
> sparse the given column qualifier is. If you trigger an OOMException, then
> the region server would abort.****
>
> >** **
>
> > For this usage specifically, though -- scanning through a single column
> qualifier for all rows -- you would be better off just doing a normal
> client side scan, ie. HTable.getScanner(). Then you will avoid the client
> timeout and potential server-side memory issues.****
>
> >** **
>
> > On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu <
> [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:****
>
> > From region server log:****
>
> >** **
>
> > 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error **
> **
>
> > making BlockReader. Closing stale ****
>
> > Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localport=**
> **
>
> > 54114]****
>
> > java.io.EOFException: Premature EOF: no length prefix available****
>
> > at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
-
Re: Regionserver goes down while endpoint executionHimanshu Vashishtha 2013-03-14, 17:45
There is no ordering guarantee for the endpoint execution, other than
that calls will be executed in parallel across all the regions.
In case you have 5 regions, then there will be 5 separate calls to
these regions. Then, you get 5 results from these regions at your
client, where you use the Callback class to aggregate the results. You
can define your ordering this aggregate class for sure.
On Thu, Mar 14, 2013 at 10:15 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> bq. provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC client
>
> If I understand you correctly, you perform batching at the client as
> described above.
> The order would be as you expected.
>
> Cheers
>
> On Thu, Mar 14, 2013 at 10:09 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
>
>> Hi,****
>>
>> It seems due to huge data the RegionServer is getting down. Now I am
>> trying to fetch the data in parts & is running fine. I need some more info
>> about the Endpoint execution:****
>>
>> ** **
>>
>> My use case is to fetch the data from HBase as per some rowkey range & to
>> render it at UI. Since endpoints are executed in parallel so I am looking
>> to use it. ****
>>
>> ** **
>>
>> **Ø **Suppose I provide the rowkey range as rowkey1 to rowkey100 in
>> endpoint RPC client & these rowkeys are distributed at 5 regions across 4
>> region servers. If I fetch 10 records at a time, do we have any way to
>> guarantee that it would come in serial order like first result would of
>> rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 & the
>> fetch would be from rowkey11 to rowkey20, irrespective of the region &
>> region servers?****
>>
>> ** **
>>
>> Regards,****
>>
>> Deepak****
>>
>> ** **
>>
>> ** **
>>
>> -----Original Message-----
>> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of
>> Himanshu Vashishtha
>> Sent: Wednesday, March 13, 2013 12:09 PM
>> To: [EMAIL PROTECTED]
>> Cc: Gary Helmling; [EMAIL PROTECTED]; lars hofhansl
>> Subject: Re: Regionserver goes down while endpoint execution
>>
>> ** **
>>
>> On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>
>> wrote:****
>>
>> > Thanks guys for assisting. I am getting OOM exception yet. I have one
>> query about Endpoints. As endpoint executes in parallel, so if I have a
>> table which is distributed at 101 regions across 5 regionserver. Would it
>> be 101 threads of endpoint executing in parallel?****
>>
>> ** **
>>
>> No and Yes.****
>>
>> ** **
>>
>> The endpoints are not processed as separate threads, they are processed as
>> just another request (via regionserver handlers). Yes, the execution will
>> be in parallel in the sense that a separate client side call will be used
>> for each of the regions that are in the range you specify.****
>>
>> ** **
>>
>> >** **
>>
>> > Regards,****
>>
>> > Deepak****
>>
>> >** **
>>
>> > From: Gary Helmling [mailto:[EMAIL PROTECTED] <[EMAIL PROTECTED]>]**
>> **
>>
>> > Sent: Tuesday, March 12, 2013 2:14 PM****
>>
>> > To: [EMAIL PROTECTED]****
>>
>> > Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]****
>>
>> > Subject: Re: Regionserver goes down while endpoint execution****
>>
>> >** **
>>
>> > To expand on what Himanshu said, your endpoint is doing an unbounded
>> scan on the region, so with a region with a lot of rows it's taking more
>> than 60 seconds to run to the region end, which is why the client side of
>> the call is timing out. In addition you're building up an in memory list
>> of all the values for that qualifier in that region, which could cause you
>> to bump into OOM issues, depending on how big your values are and how
>> sparse the given column qualifier is. If you trigger an OOMException, then
>> the region server would abort.****
>>
>> >** **
>>
>> > For this usage specifically, though -- scanning through a single column
>> qualifier for all rows -- you would be better off just doing a normal
>> client side scan, ie. HTable.getScanner(). Then you will avoid the client
>> timeout and potential server-side memory issues.****
that calls will be executed in parallel across all the regions.
In case you have 5 regions, then there will be 5 separate calls to
these regions. Then, you get 5 results from these regions at your
client, where you use the Callback class to aggregate the results. You
can define your ordering this aggregate class for sure.
On Thu, Mar 14, 2013 at 10:15 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> bq. provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC client
>
> If I understand you correctly, you perform batching at the client as
> described above.
> The order would be as you expected.
>
> Cheers
>
> On Thu, Mar 14, 2013 at 10:09 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
>
>> Hi,****
>>
>> It seems due to huge data the RegionServer is getting down. Now I am
>> trying to fetch the data in parts & is running fine. I need some more info
>> about the Endpoint execution:****
>>
>> ** **
>>
>> My use case is to fetch the data from HBase as per some rowkey range & to
>> render it at UI. Since endpoints are executed in parallel so I am looking
>> to use it. ****
>>
>> ** **
>>
>> **Ø **Suppose I provide the rowkey range as rowkey1 to rowkey100 in
>> endpoint RPC client & these rowkeys are distributed at 5 regions across 4
>> region servers. If I fetch 10 records at a time, do we have any way to
>> guarantee that it would come in serial order like first result would of
>> rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 & the
>> fetch would be from rowkey11 to rowkey20, irrespective of the region &
>> region servers?****
>>
>> ** **
>>
>> Regards,****
>>
>> Deepak****
>>
>> ** **
>>
>> ** **
>>
>> -----Original Message-----
>> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of
>> Himanshu Vashishtha
>> Sent: Wednesday, March 13, 2013 12:09 PM
>> To: [EMAIL PROTECTED]
>> Cc: Gary Helmling; [EMAIL PROTECTED]; lars hofhansl
>> Subject: Re: Regionserver goes down while endpoint execution
>>
>> ** **
>>
>> On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>
>> wrote:****
>>
>> > Thanks guys for assisting. I am getting OOM exception yet. I have one
>> query about Endpoints. As endpoint executes in parallel, so if I have a
>> table which is distributed at 101 regions across 5 regionserver. Would it
>> be 101 threads of endpoint executing in parallel?****
>>
>> ** **
>>
>> No and Yes.****
>>
>> ** **
>>
>> The endpoints are not processed as separate threads, they are processed as
>> just another request (via regionserver handlers). Yes, the execution will
>> be in parallel in the sense that a separate client side call will be used
>> for each of the regions that are in the range you specify.****
>>
>> ** **
>>
>> >** **
>>
>> > Regards,****
>>
>> > Deepak****
>>
>> >** **
>>
>> > From: Gary Helmling [mailto:[EMAIL PROTECTED] <[EMAIL PROTECTED]>]**
>> **
>>
>> > Sent: Tuesday, March 12, 2013 2:14 PM****
>>
>> > To: [EMAIL PROTECTED]****
>>
>> > Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]****
>>
>> > Subject: Re: Regionserver goes down while endpoint execution****
>>
>> >** **
>>
>> > To expand on what Himanshu said, your endpoint is doing an unbounded
>> scan on the region, so with a region with a lot of rows it's taking more
>> than 60 seconds to run to the region end, which is why the client side of
>> the call is timing out. In addition you're building up an in memory list
>> of all the values for that qualifier in that region, which could cause you
>> to bump into OOM issues, depending on how big your values are and how
>> sparse the given column qualifier is. If you trigger an OOMException, then
>> the region server would abort.****
>>
>> >** **
>>
>> > For this usage specifically, though -- scanning through a single column
>> qualifier for all rows -- you would be better off just doing a normal
>> client side scan, ie. HTable.getScanner(). Then you will avoid the client
>> timeout and potential server-side memory issues.****
-
RE: Regionserver goes down while endpoint executionAnoop Sam John 2013-03-15, 06:55
Himanshu told it clearly. To make to more clear I am adding :)
When the range of rowkeys that you are looking for spread across 5 regions, at the client side there will be 5 exec requests created and submitted to a thread pool.[HBase client side thread pool associated with HTable]
Now as per the availability of slots in this pool the operations will be requested to regions in RSs. So in different RS there can be parallel execution. Within one RS itself parallel execution on different regions also possible.
It is upto the client app to deal with the results what the Endpoint has returned. It will return a Map right? region name vs result from region. Client app need to take care about the ordering
-Anoop-
________________________________________
From: [EMAIL PROTECTED] [[EMAIL PROTECTED]] on behalf of Himanshu Vashishtha [[EMAIL PROTECTED]]
Sent: Thursday, March 14, 2013 11:15 PM
To: [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
There is no ordering guarantee for the endpoint execution, other than
that calls will be executed in parallel across all the regions.
In case you have 5 regions, then there will be 5 separate calls to
these regions. Then, you get 5 results from these regions at your
client, where you use the Callback class to aggregate the results. You
can define your ordering this aggregate class for sure.
On Thu, Mar 14, 2013 at 10:15 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> bq. provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC client
>
> If I understand you correctly, you perform batching at the client as
> described above.
> The order would be as you expected.
>
> Cheers
>
> On Thu, Mar 14, 2013 at 10:09 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
>
>> Hi,****
>>
>> It seems due to huge data the RegionServer is getting down. Now I am
>> trying to fetch the data in parts & is running fine. I need some more info
>> about the Endpoint execution:****
>>
>> ** **
>>
>> My use case is to fetch the data from HBase as per some rowkey range & to
>> render it at UI. Since endpoints are executed in parallel so I am looking
>> to use it. ****
>>
>> ** **
>>
>> **Ø **Suppose I provide the rowkey range as rowkey1 to rowkey100 in
>> endpoint RPC client & these rowkeys are distributed at 5 regions across 4
>> region servers. If I fetch 10 records at a time, do we have any way to
>> guarantee that it would come in serial order like first result would of
>> rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 & the
>> fetch would be from rowkey11 to rowkey20, irrespective of the region &
>> region servers?****
>>
>> ** **
>>
>> Regards,****
>>
>> Deepak****
>>
>> ** **
>>
>> ** **
>>
>> -----Original Message-----
>> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of
>> Himanshu Vashishtha
>> Sent: Wednesday, March 13, 2013 12:09 PM
>> To: [EMAIL PROTECTED]
>> Cc: Gary Helmling; [EMAIL PROTECTED]; lars hofhansl
>> Subject: Re: Regionserver goes down while endpoint execution
>>
>> ** **
>>
>> On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>
>> wrote:****
>>
>> > Thanks guys for assisting. I am getting OOM exception yet. I have one
>> query about Endpoints. As endpoint executes in parallel, so if I have a
>> table which is distributed at 101 regions across 5 regionserver. Would it
>> be 101 threads of endpoint executing in parallel?****
>>
>> ** **
>>
>> No and Yes.****
>>
>> ** **
>>
>> The endpoints are not processed as separate threads, they are processed as
>> just another request (via regionserver handlers). Yes, the execution will
>> be in parallel in the sense that a separate client side call will be used
>> for each of the regions that are in the range you specify.****
>>
>> ** **
>>
>> >** **
>>
>> > Regards,****
>>
>> > Deepak****
>>
>> >** **
>>
>> > From: Gary Helmling [mailto:[EMAIL PROTECTED] <[EMAIL PROTECTED]>]**
>> **
>>
>> > Sent: Tuesday, March 12, 2013 2:14 PM****
>>
>> > To: [EMAIL PROTECTED]****
When the range of rowkeys that you are looking for spread across 5 regions, at the client side there will be 5 exec requests created and submitted to a thread pool.[HBase client side thread pool associated with HTable]
Now as per the availability of slots in this pool the operations will be requested to regions in RSs. So in different RS there can be parallel execution. Within one RS itself parallel execution on different regions also possible.
It is upto the client app to deal with the results what the Endpoint has returned. It will return a Map right? region name vs result from region. Client app need to take care about the ordering
-Anoop-
________________________________________
From: [EMAIL PROTECTED] [[EMAIL PROTECTED]] on behalf of Himanshu Vashishtha [[EMAIL PROTECTED]]
Sent: Thursday, March 14, 2013 11:15 PM
To: [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
There is no ordering guarantee for the endpoint execution, other than
that calls will be executed in parallel across all the regions.
In case you have 5 regions, then there will be 5 separate calls to
these regions. Then, you get 5 results from these regions at your
client, where you use the Callback class to aggregate the results. You
can define your ordering this aggregate class for sure.
On Thu, Mar 14, 2013 at 10:15 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> bq. provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC client
>
> If I understand you correctly, you perform batching at the client as
> described above.
> The order would be as you expected.
>
> Cheers
>
> On Thu, Mar 14, 2013 at 10:09 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
>
>> Hi,****
>>
>> It seems due to huge data the RegionServer is getting down. Now I am
>> trying to fetch the data in parts & is running fine. I need some more info
>> about the Endpoint execution:****
>>
>> ** **
>>
>> My use case is to fetch the data from HBase as per some rowkey range & to
>> render it at UI. Since endpoints are executed in parallel so I am looking
>> to use it. ****
>>
>> ** **
>>
>> **Ø **Suppose I provide the rowkey range as rowkey1 to rowkey100 in
>> endpoint RPC client & these rowkeys are distributed at 5 regions across 4
>> region servers. If I fetch 10 records at a time, do we have any way to
>> guarantee that it would come in serial order like first result would of
>> rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 & the
>> fetch would be from rowkey11 to rowkey20, irrespective of the region &
>> region servers?****
>>
>> ** **
>>
>> Regards,****
>>
>> Deepak****
>>
>> ** **
>>
>> ** **
>>
>> -----Original Message-----
>> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of
>> Himanshu Vashishtha
>> Sent: Wednesday, March 13, 2013 12:09 PM
>> To: [EMAIL PROTECTED]
>> Cc: Gary Helmling; [EMAIL PROTECTED]; lars hofhansl
>> Subject: Re: Regionserver goes down while endpoint execution
>>
>> ** **
>>
>> On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>
>> wrote:****
>>
>> > Thanks guys for assisting. I am getting OOM exception yet. I have one
>> query about Endpoints. As endpoint executes in parallel, so if I have a
>> table which is distributed at 101 regions across 5 regionserver. Would it
>> be 101 threads of endpoint executing in parallel?****
>>
>> ** **
>>
>> No and Yes.****
>>
>> ** **
>>
>> The endpoints are not processed as separate threads, they are processed as
>> just another request (via regionserver handlers). Yes, the execution will
>> be in parallel in the sense that a separate client side call will be used
>> for each of the regions that are in the range you specify.****
>>
>> ** **
>>
>> >** **
>>
>> > Regards,****
>>
>> > Deepak****
>>
>> >** **
>>
>> > From: Gary Helmling [mailto:[EMAIL PROTECTED] <[EMAIL PROTECTED]>]**
>> **
>>
>> > Sent: Tuesday, March 12, 2013 2:14 PM****
>>
>> > To: [EMAIL PROTECTED]****
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-20, 07:41
Hi Anoop,
Quite inspired by your coprocessor secondary indexing document & trying to implement one, for better response :)
The coprocessor executes for some time, but later on (say after 400-500 inserts) it gives IndexOutofBounds exception.
The stack trace is
2013-03-20 02:40:42,074 INFO com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor: Counter: 2408
2013-03-20 02:40:42,098 INFO com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor: Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:/var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:vm-ab1f-dd21.nam.nsroot.net:
2013-03-20 02:40:42,098 INFO com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor: Counter: 2410
2013-03-20 02:40:42,122 ERROR org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d' from environment because it threw: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.remove(ArrayList.java:387)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
at com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
If I synchronize on current object in the postPut then it works fine? If I use some counter valriable in the postput then it seems the counter misses some value just before this exception. As you can see above 2409 is missing. But if I synchronize the postPut block then it executes fine & the counter is also in serial order. But I think using synchronize would slow down the insertion in secondary index. Could you guide me the exact reason for this?
Here is the complete code for postPut
public void postPut(final ObserverContext<RegionCoprocessorEnvironment> e,
final Put put, final WALEdit edit, final boolean writeToWAL) throws IOException {
List<KeyValue> hostName = put.get(Bytes.toBytes("sysInfo"), Bytes.toBytes("hostName"));
List<KeyValue> logFilePath = put.get(Bytes.toBytes("content"), Bytes.toBytes("logFilePath"));
List<KeyValue> logFileName = put.get(Bytes.toBytes("content"), Bytes.toBytes("logFileName"));
if(hostName.size() > 0 && logFilePath.size() > 0 && logFileName.size() >0){
byte[] hostNameVal = hostName.get(0).getValue();
byte[] logFilePathVal = logFilePath.get(0).getValue();
byte[] logFileNameVal = logFileName.get(0).getValue();
byte[] rowKeyBody = hostName.get(0).getRow();
synchronized(this){
byte[] rowKey = (
Bytes.toString(hostNameVal) + ":" + Bytes.toString(logFilePathVal) + Bytes.toString(logFileNameVal)
+ LOG_INDEX_DELIM + Bytes.toString(rowKeyBody)).getBytes();
logger.debug("Row Key Secondary Index: "+Bytes.toString(rowKey));
Put putInIndexTable = new Put(rowKey);
putInIndexTable.add(LOG_INDEX_FAMILY_CORE.getBytes(), LOG_INDEX_ELFLOG_RK.getBytes(), Bytes.toBytes(dummyRowkey));
dummyRowkey++;
this.table.put(putInIndexTable);
logger.info("Added in "+tableName+": "+Bytes.toString(rowKey));
logger.info("Counter: "+dummyRowkey);
}
}else{
logger.error("Size of hostName"+hostName.size());
logger.error("Size of logFilePath"+logFilePath.size());
logger.error("Size of logFileName"+logFileName.size());
logger.error("Size of logFileBody"+hostName.size());
}
}
Regards,
Deepak
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Friday, March 15, 2013 2:56 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Himanshu told it clearly. To make to more clear I am adding :)
When the range of rowkeys that you are looking for spread across 5 regions, at the client side there will be 5 exec requests created and submitted to a thread pool.[HBase client side thread pool associated with HTable] Now as per the availability of slots in this pool the operations will be requested to regions in RSs. So in different RS there can be parallel execution. Within one RS itself parallel execution on different regions also possible.
It is upto the client app to deal with the results what the Endpoint has returned. It will return a Map right? region name vs result from region. Client app need to take care about the ordering
-Anoop-
________________________________________
From: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]> [[EMAIL PROTECTED]] on behalf of Himanshu Vashishtha [[EMAIL PROTECTED]]
Sent: Thursday, March 14, 2013 11:15 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
There is no ordering guarantee for the endpoint execution, other than that calls will be executed in parallel across all the regions.
In case you have 5 regions, then there will be 5 separate calls to these regions. Then, you get 5 results from these regions at your client, where you use the Callback class to aggregate the results. You can define your ordering this aggregate class for sure.
On Thu, Mar 14, 2013 at 10:15 AM, Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Quite inspired by your coprocessor secondary indexing document & trying to implement one, for better response :)
The coprocessor executes for some time, but later on (say after 400-500 inserts) it gives IndexOutofBounds exception.
The stack trace is
2013-03-20 02:40:42,074 INFO com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor: Counter: 2408
2013-03-20 02:40:42,098 INFO com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor: Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:/var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:vm-ab1f-dd21.nam.nsroot.net:
2013-03-20 02:40:42,098 INFO com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor: Counter: 2410
2013-03-20 02:40:42,122 ERROR org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d' from environment because it threw: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.remove(ArrayList.java:387)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
at com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
If I synchronize on current object in the postPut then it works fine? If I use some counter valriable in the postput then it seems the counter misses some value just before this exception. As you can see above 2409 is missing. But if I synchronize the postPut block then it executes fine & the counter is also in serial order. But I think using synchronize would slow down the insertion in secondary index. Could you guide me the exact reason for this?
Here is the complete code for postPut
public void postPut(final ObserverContext<RegionCoprocessorEnvironment> e,
final Put put, final WALEdit edit, final boolean writeToWAL) throws IOException {
List<KeyValue> hostName = put.get(Bytes.toBytes("sysInfo"), Bytes.toBytes("hostName"));
List<KeyValue> logFilePath = put.get(Bytes.toBytes("content"), Bytes.toBytes("logFilePath"));
List<KeyValue> logFileName = put.get(Bytes.toBytes("content"), Bytes.toBytes("logFileName"));
if(hostName.size() > 0 && logFilePath.size() > 0 && logFileName.size() >0){
byte[] hostNameVal = hostName.get(0).getValue();
byte[] logFilePathVal = logFilePath.get(0).getValue();
byte[] logFileNameVal = logFileName.get(0).getValue();
byte[] rowKeyBody = hostName.get(0).getRow();
synchronized(this){
byte[] rowKey = (
Bytes.toString(hostNameVal) + ":" + Bytes.toString(logFilePathVal) + Bytes.toString(logFileNameVal)
+ LOG_INDEX_DELIM + Bytes.toString(rowKeyBody)).getBytes();
logger.debug("Row Key Secondary Index: "+Bytes.toString(rowKey));
Put putInIndexTable = new Put(rowKey);
putInIndexTable.add(LOG_INDEX_FAMILY_CORE.getBytes(), LOG_INDEX_ELFLOG_RK.getBytes(), Bytes.toBytes(dummyRowkey));
dummyRowkey++;
this.table.put(putInIndexTable);
logger.info("Added in "+tableName+": "+Bytes.toString(rowKey));
logger.info("Counter: "+dummyRowkey);
}
}else{
logger.error("Size of hostName"+hostName.size());
logger.error("Size of logFilePath"+logFilePath.size());
logger.error("Size of logFileName"+logFileName.size());
logger.error("Size of logFileBody"+hostName.size());
}
}
Regards,
Deepak
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Friday, March 15, 2013 2:56 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Himanshu told it clearly. To make to more clear I am adding :)
When the range of rowkeys that you are looking for spread across 5 regions, at the client side there will be 5 exec requests created and submitted to a thread pool.[HBase client side thread pool associated with HTable] Now as per the availability of slots in this pool the operations will be requested to regions in RSs. So in different RS there can be parallel execution. Within one RS itself parallel execution on different regions also possible.
It is upto the client app to deal with the results what the Endpoint has returned. It will return a Map right? region name vs result from region. Client app need to take care about the ordering
-Anoop-
________________________________________
From: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]> [[EMAIL PROTECTED]] on behalf of Himanshu Vashishtha [[EMAIL PROTECTED]]
Sent: Thursday, March 14, 2013 11:15 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
There is no ordering guarantee for the endpoint execution, other than that calls will be executed in parallel across all the regions.
In case you have 5 regions, then there will be 5 separate calls to these regions. Then, you get 5 results from these regions at your client, where you use the Callback class to aggregate the results. You can define your ordering this aggregate class for sure.
On Thu, Mar 14, 2013 at 10:15 AM, Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
-
Re: Regionserver goes down while endpoint executionramkrishna vasudevan 2013-03-20, 08:01
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
> Index: 1, Size: 1
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>
> at java.util.ArrayList.remove(ArrayList.java:387)
>
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
>
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
>
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
>
> at
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
>
> at
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
>
>
>
> If I synchronize on current object in the postPut then it works fine? If I
> use some counter valriable in the postput then it seems the counter misses
> some value just before this exception. As you can see above 2409 is
> missing. But if I synchronize the postPut block then it executes fine & the
> counter is also in serial order. But I think using synchronize would slow
> down the insertion in secondary index. Could you guide me the exact reason
> for this?
>
>
>
> Here is the complete code for postPut
>
>
>
> public void postPut(final ObserverContext<RegionCoprocessorEnvironment> e,
>
> final Put put, final WALEdit edit, final boolean
> writeToWAL) throws IOException {
>
>
>
> List<KeyValue> hostName = put.get(Bytes.toBytes("sysInfo"),
> Bytes.toBytes("hostName"));
>
> List<KeyValue> logFilePath = put.get(Bytes.toBytes("content"),
> Bytes.toBytes("logFilePath"));
>
> List<KeyValue> logFileName = put.get(Bytes.toBytes("content"),
> Bytes.toBytes("logFileName"));
>
>
>
> if(hostName.size() > 0 && logFilePath.size() > 0 &&
> logFileName.size() >0){
>
> byte[] hostNameVal = hostName.get(0).getValue();
>
> byte[] logFilePathVal = logFilePath.get(0).getValue();
>
> byte[] logFileNameVal = logFileName.get(0).getValue();
>
> byte[] rowKeyBody = hostName.get(0).getRow();
>
> synchronized(this){
>
>
>
>
>
> byte[] rowKey = (
>
> Bytes.toString(hostNameVal) + ":" +
> Bytes.toString(logFilePathVal) + Bytes.toString(logFileNameVal)
>
> + LOG_INDEX_DELIM +
> Bytes.toString(rowKeyBody)).getBytes();
>
>
>
> logger.debug("Row Key Secondary Index:
> "+Bytes.toString(rowKey));
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
> Index: 1, Size: 1
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>
> at java.util.ArrayList.remove(ArrayList.java:387)
>
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
>
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
>
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
>
> at
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
>
> at
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
>
>
>
> If I synchronize on current object in the postPut then it works fine? If I
> use some counter valriable in the postput then it seems the counter misses
> some value just before this exception. As you can see above 2409 is
> missing. But if I synchronize the postPut block then it executes fine & the
> counter is also in serial order. But I think using synchronize would slow
> down the insertion in secondary index. Could you guide me the exact reason
> for this?
>
>
>
> Here is the complete code for postPut
>
>
>
> public void postPut(final ObserverContext<RegionCoprocessorEnvironment> e,
>
> final Put put, final WALEdit edit, final boolean
> writeToWAL) throws IOException {
>
>
>
> List<KeyValue> hostName = put.get(Bytes.toBytes("sysInfo"),
> Bytes.toBytes("hostName"));
>
> List<KeyValue> logFilePath = put.get(Bytes.toBytes("content"),
> Bytes.toBytes("logFilePath"));
>
> List<KeyValue> logFileName = put.get(Bytes.toBytes("content"),
> Bytes.toBytes("logFileName"));
>
>
>
> if(hostName.size() > 0 && logFilePath.size() > 0 &&
> logFileName.size() >0){
>
> byte[] hostNameVal = hostName.get(0).getValue();
>
> byte[] logFilePathVal = logFilePath.get(0).getValue();
>
> byte[] logFileNameVal = logFileName.get(0).getValue();
>
> byte[] rowKeyBody = hostName.get(0).getRow();
>
> synchronized(this){
>
>
>
>
>
> byte[] rowKey = (
>
> Bytes.toString(hostNameVal) + ":" +
> Bytes.toString(logFilePathVal) + Bytes.toString(logFileNameVal)
>
> + LOG_INDEX_DELIM +
> Bytes.toString(rowKeyBody)).getBytes();
>
>
>
> logger.debug("Row Key Secondary Index:
> "+Bytes.toString(rowKey));
-
RE: Regionserver goes down while endpoint executionAnoop Sam John 2013-03-20, 08:36
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
> Index: 1, Size: 1
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>
> at java.util.ArrayList.remove(ArrayList.java:387)
>
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
>
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
>
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
>
> at
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
>
> at
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
>
>
>
> If I synchronize on current object in the postPut then it works fine? If I
> use some counter valriable in the postput then it seems the counter misses
> some value just before this exception. As you can see above 2409 is
> missing. But if I synchronize the postPut block then it executes fine & the
> counter is also in serial order. But I think using synchronize would slow
> down the insertion in secondary index. Could you guide me the exact reason
> for this?
>
>
>
> Here is the complete code for postPut
>
>
>
> public void postPut(final ObserverContext<RegionCoprocessorEnvironment> e,
>
> final Put put, final WALEdit edit, final boolean
> writeToWAL) throws IOException {
>
>
>
> List<KeyValue> hostName = put.get(Bytes.toBytes("sysInfo"),
> Bytes.toBytes("hostName"));
>
> List<KeyValue> logFilePath = put.get(Bytes.toBytes("content"),
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
> Index: 1, Size: 1
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>
> at java.util.ArrayList.remove(ArrayList.java:387)
>
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
>
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
>
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
>
> at
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
>
> at
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
>
>
>
> If I synchronize on current object in the postPut then it works fine? If I
> use some counter valriable in the postput then it seems the counter misses
> some value just before this exception. As you can see above 2409 is
> missing. But if I synchronize the postPut block then it executes fine & the
> counter is also in serial order. But I think using synchronize would slow
> down the insertion in secondary index. Could you guide me the exact reason
> for this?
>
>
>
> Here is the complete code for postPut
>
>
>
> public void postPut(final ObserverContext<RegionCoprocessorEnvironment> e,
>
> final Put put, final WALEdit edit, final boolean
> writeToWAL) throws IOException {
>
>
>
> List<KeyValue> hostName = put.get(Bytes.toBytes("sysInfo"),
> Bytes.toBytes("hostName"));
>
> List<KeyValue> logFilePath = put.get(Bytes.toBytes("content"),
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-20, 12:44
Hi,
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
> Index: 1, Size: 1
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>
> at java.util.ArrayList.remove(ArrayList.java:387)
>
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
>
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
>
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
>
> at
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
>
> at
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
>
>
>
> If I synchronize on current object in the postPut then it works fine? If I
> use some counter valriable in the postput then it seems the counter misses
> some value just before this exception. As you can see above 2409 is
> missing. But if I synchronize the postPut block then it executes fine & the
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
> Index: 1, Size: 1
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>
> at java.util.ArrayList.remove(ArrayList.java:387)
>
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
>
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
>
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
>
> at
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
>
> at
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
>
>
>
> If I synchronize on current object in the postPut then it works fine? If I
> use some counter valriable in the postput then it seems the counter misses
> some value just before this exception. As you can see above 2409 is
> missing. But if I synchronize the postPut block then it executes fine & the
-
RE: Regionserver goes down while endpoint executionAnoop Sam John 2013-03-20, 12:58
What is this doing?
HbaseTableAccess.getInstance().connectTable(tableName);
Can you take a look at HTablePool?
-Anoop-
________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 6:14 PM
To: '[EMAIL PROTECTED]'; Anoop Sam John; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Hi,
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
> Index: 1, Size: 1
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>
> at java.util.ArrayList.remove(ArrayList.java:387)
>
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
>
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
>
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
>
> at
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
>
> at
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
HbaseTableAccess.getInstance().connectTable(tableName);
Can you take a look at HTablePool?
-Anoop-
________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 6:14 PM
To: '[EMAIL PROTECTED]'; Anoop Sam John; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Hi,
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
> Index: 1, Size: 1
>
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>
> at java.util.ArrayList.remove(ArrayList.java:387)
>
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:960)
>
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826)
>
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801)
>
> at
> org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:394)
>
> at
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor.postPut(SecondaryIndexCoprocessor.java:91)
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-20, 13:18
Hi,
It is a custom class which is internally using HTablePool. The getInstance() is calling the private constructor which has following code:
private HbaseTableAccess(){
conf = HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum", "zookeeper-quorum-urls");
htablePool = new HTablePool(conf, 100);
}
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 8:59 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
What is this doing?
HbaseTableAccess.getInstance().connectTable(tableName);
Can you take a look at HTablePool?
-Anoop-
________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 6:14 PM
To: '[EMAIL PROTECTED]'; Anoop Sam John; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Hi,
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
It is a custom class which is internally using HTablePool. The getInstance() is calling the private constructor which has following code:
private HbaseTableAccess(){
conf = HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum", "zookeeper-quorum-urls");
htablePool = new HTablePool(conf, 100);
}
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 8:59 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
What is this doing?
HbaseTableAccess.getInstance().connectTable(tableName);
Can you take a look at HTablePool?
-Anoop-
________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 6:14 PM
To: '[EMAIL PROTECTED]'; Anoop Sam John; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Hi,
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-25, 16:53
Hi Anoop,
Could you please provide some pointers. If we have a regionbase coprocessor which has some set of operation implemented in its postPut implementation.
If I have say 1000 invocations of postPut per seconds, are those calls would be blocking? I mean, the second call will only start after the completion of first call of postPut, or it would execute independently?
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 8:59 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
What is this doing?
HbaseTableAccess.getInstance().connectTable(tableName);
Can you take a look at HTablePool?
-Anoop-
________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 6:14 PM
To: '[EMAIL PROTECTED]'; Anoop Sam John; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Hi,
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
Could you please provide some pointers. If we have a regionbase coprocessor which has some set of operation implemented in its postPut implementation.
If I have say 1000 invocations of postPut per seconds, are those calls would be blocking? I mean, the second call will only start after the completion of first call of postPut, or it would execute independently?
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 8:59 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
What is this doing?
HbaseTableAccess.getInstance().connectTable(tableName);
Can you take a look at HTablePool?
-Anoop-
________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 6:14 PM
To: '[EMAIL PROTECTED]'; Anoop Sam John; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Hi,
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2408
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Added in elf_log_index: vm-ab1f-dd21.nam.nsroot.net:
> /var/log/flume/flume-root-node-vm-ab1f-dd21.log::153299:1363758015261:3913805817870658:
> vm-ab1f-dd21.nam.nsroot.net:
>
> 2013-03-20 02:40:42,098 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
> Counter: 2410
>
> 2013-03-20 02:40:42,122 ERROR
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Removing coprocessor
> 'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@2f86780d'
> from environment because it threw: java.lang.IndexOutOfBoundsException:
-
RE: Regionserver goes down while endpoint executionAnoop Sam John 2013-03-26, 06:20
Deepak
When you say 1000 invocations, you mean you are having a put(List) with 1000 items? In the region level when it deals with a batch of Puts, it calls postPut() for each of the Put one after the other. In the same handler thread which is doing the write. So yes one need to wait for the other. Otherwise you need to handle within the implementation of having a thread pool or so.
I would say rethink on doing HTable#put() from a CP hook.
-Anoop-
________________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Monday, March 25, 2013 10:23 PM
To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop Sam John
Subject: RE: Regionserver goes down while endpoint execution
Hi Anoop,
Could you please provide some pointers. If we have a regionbase coprocessor which has some set of operation implemented in its postPut implementation.
If I have say 1000 invocations of postPut per seconds, are those calls would be blocking? I mean, the second call will only start after the completion of first call of postPut, or it would execute independently?
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 8:59 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
What is this doing?
HbaseTableAccess.getInstance().connectTable(tableName);
Can you take a look at HTablePool?
-Anoop-
________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 6:14 PM
To: '[EMAIL PROTECTED]'; Anoop Sam John; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Hi,
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
When you say 1000 invocations, you mean you are having a put(List) with 1000 items? In the region level when it deals with a batch of Puts, it calls postPut() for each of the Put one after the other. In the same handler thread which is doing the write. So yes one need to wait for the other. Otherwise you need to handle within the implementation of having a thread pool or so.
I would say rethink on doing HTable#put() from a CP hook.
-Anoop-
________________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Monday, March 25, 2013 10:23 PM
To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop Sam John
Subject: RE: Regionserver goes down while endpoint execution
Hi Anoop,
Could you please provide some pointers. If we have a regionbase coprocessor which has some set of operation implemented in its postPut implementation.
If I have say 1000 invocations of postPut per seconds, are those calls would be blocking? I mean, the second call will only start after the completion of first call of postPut, or it would execute independently?
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 8:59 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
What is this doing?
HbaseTableAccess.getInstance().connectTable(tableName);
Can you take a look at HTablePool?
-Anoop-
________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 6:14 PM
To: '[EMAIL PROTECTED]'; Anoop Sam John; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Hi,
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
> Hi Anoop,
>
> Quite inspired by your coprocessor secondary indexing document & trying to
> implement one, for better response :)
>
>
>
> The coprocessor executes for some time, but later on (say after 400-500
> inserts) it gives IndexOutofBounds exception.
>
>
>
> The stack trace is
>
> 2013-03-20 02:40:42,074 INFO
> com.citi.sponge.hbase.coprocessor.secondaryindex.SecondaryIndexCoprocessor:
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-26, 07:27
Thanks Anoop for assisting.
I don't have Put(List). All those calls are independent of each other. Let me put it this way. Suppose I have 1000 insertions/seconds on a table, (each insert is independent & not coming in a list). Would it be a performance impact to use CP to update secondary index?
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 26, 2013 2:20 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Deepak
When you say 1000 invocations, you mean you are having a put(List) with 1000 items? In the region level when it deals with a batch of Puts, it calls postPut() for each of the Put one after the other. In the same handler thread which is doing the write. So yes one need to wait for the other. Otherwise you need to handle within the implementation of having a thread pool or so.
I would say rethink on doing HTable#put() from a CP hook.
-Anoop-
________________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Monday, March 25, 2013 10:23 PM
To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop Sam John
Subject: RE: Regionserver goes down while endpoint execution
Hi Anoop,
Could you please provide some pointers. If we have a regionbase coprocessor which has some set of operation implemented in its postPut implementation.
If I have say 1000 invocations of postPut per seconds, are those calls would be blocking? I mean, the second call will only start after the completion of first call of postPut, or it would execute independently?
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 8:59 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
What is this doing?
HbaseTableAccess.getInstance().connectTable(tableName);
Can you take a look at HTablePool?
-Anoop-
________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 6:14 PM
To: '[EMAIL PROTECTED]'; Anoop Sam John; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Hi,
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
I don't have Put(List). All those calls are independent of each other. Let me put it this way. Suppose I have 1000 insertions/seconds on a table, (each insert is independent & not coming in a list). Would it be a performance impact to use CP to update secondary index?
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 26, 2013 2:20 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Deepak
When you say 1000 invocations, you mean you are having a put(List) with 1000 items? In the region level when it deals with a batch of Puts, it calls postPut() for each of the Put one after the other. In the same handler thread which is doing the write. So yes one need to wait for the other. Otherwise you need to handle within the implementation of having a thread pool or so.
I would say rethink on doing HTable#put() from a CP hook.
-Anoop-
________________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Monday, March 25, 2013 10:23 PM
To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop Sam John
Subject: RE: Regionserver goes down while endpoint execution
Hi Anoop,
Could you please provide some pointers. If we have a regionbase coprocessor which has some set of operation implemented in its postPut implementation.
If I have say 1000 invocations of postPut per seconds, are those calls would be blocking? I mean, the second call will only start after the completion of first call of postPut, or it would execute independently?
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 8:59 AM
To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
What is this doing?
HbaseTableAccess.getInstance().connectTable(tableName);
Can you take a look at HTablePool?
-Anoop-
________________________________
From: Kumar, Deepak8 [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 6:14 PM
To: '[EMAIL PROTECTED]'; Anoop Sam John; '[EMAIL PROTECTED]'
Subject: RE: Regionserver goes down while endpoint execution
Hi,
Even when I moved table = HbaseTableAccess.getInstance().connectTable(tableName); to postPut method of coprocessor, I am getting the same exception.
Regards,
Deepak
-----Original Message-----
From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 4:37 AM
To: [EMAIL PROTECTED]
Subject: RE: Regionserver goes down while endpoint execution
Hi Deepak
Yes Ram told it correctly regarding the thread safety issue with HTable.
I can see you issue HTable#put() to write to index table and this call you are making from a CP.
Our design was around making the actual table region and corresponding index table region collocated in the same RS. In our impl we are not calling HTable#put() from CP. We get the index region and directly call put on that.
In some mail thread Andrew Purtell explained it nicely that doing a client based call like this from CP is an anti pattern. Think of avoiding this if you are going with CP based impl. If you are doing with the current way or you need a global ordered indexing, IMHO think of managing the index from the client side only rather than CP.
-Anoop-
________________________________________
From: ramkrishna vasudevan [[EMAIL PROTECTED]]
Sent: Wednesday, March 20, 2013 1:31 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Hi Deepak
"this.table.put(putInIndexTable);""
I think this one should be the problem.
Your table is at the instance level. See the documentation of HTable.
"<p>This class is not thread safe for reads nor write."
So if you try creating a new HTable every time this problem should be
avoided.
Regards
Ram
On Wed, Mar 20, 2013 at 1:11 PM, Kumar, Deepak8 <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>wrote:
-
Re: Regionserver goes down while endpoint executionAdrien Mogenet 2013-03-26, 07:42
Sure, and it might be costly if your CP is building secondary index through
`Put' to external tables/regions. Each of your RPC call will generate
another RPC call. Also, take note that your client could get an unexpected
behavior; it might be blocked waiting for RPC answer just because of your
CP.
As Anoop said, rethink on doing such things from your CP hook. Otherwise,
think about implementing asynchronous `Put' using a thread but it might
lead in some headach (ensure - eventually - consistency, handle
reconections, etc.)
On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Thanks Anoop for assisting.
>
> I don't have Put(List). All those calls are independent of each other. Let
> me put it this way. Suppose I have 1000 insertions/seconds on a table,
> (each insert is independent & not coming in a list). Would it be a
> performance impact to use CP to update secondary index?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 2:20 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
> Deepak
> When you say 1000 invocations, you mean you are having a put(List)
> with 1000 items? In the region level when it deals with a batch of Puts,
> it calls postPut() for each of the Put one after the other. In the same
> handler thread which is doing the write. So yes one need to wait for the
> other. Otherwise you need to handle within the implementation of having a
> thread pool or so.
>
> I would say rethink on doing HTable#put() from a CP hook.
>
> -Anoop-
>
> ________________________________________
> From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> Sent: Monday, March 25, 2013 10:23 PM
> To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop
> Sam John
> Subject: RE: Regionserver goes down while endpoint execution
>
> Hi Anoop,
>
> Could you please provide some pointers. If we have a regionbase
> coprocessor which has some set of operation implemented in its postPut
> implementation.
>
> If I have say 1000 invocations of postPut per seconds, are those calls
> would be blocking? I mean, the second call will only start after the
> completion of first call of postPut, or it would execute independently?
>
> Regards,
> Deepak
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 8:59 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
> What is this doing?
>
> HbaseTableAccess.getInstance().connectTable(tableName);
>
>
>
> Can you take a look at HTablePool?
>
>
>
> -Anoop-
>
>
>
> ________________________________
> From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 6:14 PM
> To: '[EMAIL PROTECTED]'; Anoop Sam John; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
>
> Hi,
>
> Even when I moved table > HbaseTableAccess.getInstance().connectTable(tableName); to postPut method
> of coprocessor, I am getting the same exception.
>
>
>
> Regards,
>
> Deepak
>
>
>
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 4:37 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Regionserver goes down while endpoint execution
>
>
>
> Hi Deepak
>
> Yes Ram told it correctly regarding the thread
> safety issue with HTable.
>
> I can see you issue HTable#put() to write to index table and this call you
> are making from a CP.
>
> Our design was around making the actual table region and corresponding
> index table region collocated in the same RS. In our impl we are not
> calling HTable#put() from CP. We get the index region and directly call put
> on that.
>
>
>
> In some mail thread Andrew Purtell explained it nicely that doing a client
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
`Put' to external tables/regions. Each of your RPC call will generate
another RPC call. Also, take note that your client could get an unexpected
behavior; it might be blocked waiting for RPC answer just because of your
CP.
As Anoop said, rethink on doing such things from your CP hook. Otherwise,
think about implementing asynchronous `Put' using a thread but it might
lead in some headach (ensure - eventually - consistency, handle
reconections, etc.)
On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Thanks Anoop for assisting.
>
> I don't have Put(List). All those calls are independent of each other. Let
> me put it this way. Suppose I have 1000 insertions/seconds on a table,
> (each insert is independent & not coming in a list). Would it be a
> performance impact to use CP to update secondary index?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 2:20 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
> Deepak
> When you say 1000 invocations, you mean you are having a put(List)
> with 1000 items? In the region level when it deals with a batch of Puts,
> it calls postPut() for each of the Put one after the other. In the same
> handler thread which is doing the write. So yes one need to wait for the
> other. Otherwise you need to handle within the implementation of having a
> thread pool or so.
>
> I would say rethink on doing HTable#put() from a CP hook.
>
> -Anoop-
>
> ________________________________________
> From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> Sent: Monday, March 25, 2013 10:23 PM
> To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop
> Sam John
> Subject: RE: Regionserver goes down while endpoint execution
>
> Hi Anoop,
>
> Could you please provide some pointers. If we have a regionbase
> coprocessor which has some set of operation implemented in its postPut
> implementation.
>
> If I have say 1000 invocations of postPut per seconds, are those calls
> would be blocking? I mean, the second call will only start after the
> completion of first call of postPut, or it would execute independently?
>
> Regards,
> Deepak
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 8:59 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
> What is this doing?
>
> HbaseTableAccess.getInstance().connectTable(tableName);
>
>
>
> Can you take a look at HTablePool?
>
>
>
> -Anoop-
>
>
>
> ________________________________
> From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 6:14 PM
> To: '[EMAIL PROTECTED]'; Anoop Sam John; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
>
> Hi,
>
> Even when I moved table > HbaseTableAccess.getInstance().connectTable(tableName); to postPut method
> of coprocessor, I am getting the same exception.
>
>
>
> Regards,
>
> Deepak
>
>
>
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 4:37 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Regionserver goes down while endpoint execution
>
>
>
> Hi Deepak
>
> Yes Ram told it correctly regarding the thread
> safety issue with HTable.
>
> I can see you issue HTable#put() to write to index table and this call you
> are making from a CP.
>
> Our design was around making the actual table region and corresponding
> index table region collocated in the same RS. In our impl we are not
> calling HTable#put() from CP. We get the index region and directly call put
> on that.
>
>
>
> In some mail thread Andrew Purtell explained it nicely that doing a client
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-26, 08:27
Hi Adrien,
>From CP hook, do we mean EndPoints implementation? Do we have any document for asynchronous put implementations? A simple example would also be quite better to start with.
Regards,
Deepak
-----Original Message-----
From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 26, 2013 3:42 AM
To: [EMAIL PROTECTED]
Cc: Anoop Sam John; [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
Sure, and it might be costly if your CP is building secondary index through
`Put' to external tables/regions. Each of your RPC call will generate
another RPC call. Also, take note that your client could get an unexpected
behavior; it might be blocked waiting for RPC answer just because of your
CP.
As Anoop said, rethink on doing such things from your CP hook. Otherwise,
think about implementing asynchronous `Put' using a thread but it might
lead in some headach (ensure - eventually - consistency, handle
reconections, etc.)
On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Thanks Anoop for assisting.
>
> I don't have Put(List). All those calls are independent of each other. Let
> me put it this way. Suppose I have 1000 insertions/seconds on a table,
> (each insert is independent & not coming in a list). Would it be a
> performance impact to use CP to update secondary index?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 2:20 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
> Deepak
> When you say 1000 invocations, you mean you are having a put(List)
> with 1000 items? In the region level when it deals with a batch of Puts,
> it calls postPut() for each of the Put one after the other. In the same
> handler thread which is doing the write. So yes one need to wait for the
> other. Otherwise you need to handle within the implementation of having a
> thread pool or so.
>
> I would say rethink on doing HTable#put() from a CP hook.
>
> -Anoop-
>
> ________________________________________
> From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> Sent: Monday, March 25, 2013 10:23 PM
> To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop
> Sam John
> Subject: RE: Regionserver goes down while endpoint execution
>
> Hi Anoop,
>
> Could you please provide some pointers. If we have a regionbase
> coprocessor which has some set of operation implemented in its postPut
> implementation.
>
> If I have say 1000 invocations of postPut per seconds, are those calls
> would be blocking? I mean, the second call will only start after the
> completion of first call of postPut, or it would execute independently?
>
> Regards,
> Deepak
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 8:59 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
> What is this doing?
>
> HbaseTableAccess.getInstance().connectTable(tableName);
>
>
>
> Can you take a look at HTablePool?
>
>
>
> -Anoop-
>
>
>
> ________________________________
> From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 6:14 PM
> To: '[EMAIL PROTECTED]'; Anoop Sam John; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
>
> Hi,
>
> Even when I moved table > HbaseTableAccess.getInstance().connectTable(tableName); to postPut method
> of coprocessor, I am getting the same exception.
>
>
>
> Regards,
>
> Deepak
>
>
>
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 4:37 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Regionserver goes down while endpoint execution
>
>
>
> Hi Deepak
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
>From CP hook, do we mean EndPoints implementation? Do we have any document for asynchronous put implementations? A simple example would also be quite better to start with.
Regards,
Deepak
-----Original Message-----
From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 26, 2013 3:42 AM
To: [EMAIL PROTECTED]
Cc: Anoop Sam John; [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
Sure, and it might be costly if your CP is building secondary index through
`Put' to external tables/regions. Each of your RPC call will generate
another RPC call. Also, take note that your client could get an unexpected
behavior; it might be blocked waiting for RPC answer just because of your
CP.
As Anoop said, rethink on doing such things from your CP hook. Otherwise,
think about implementing asynchronous `Put' using a thread but it might
lead in some headach (ensure - eventually - consistency, handle
reconections, etc.)
On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Thanks Anoop for assisting.
>
> I don't have Put(List). All those calls are independent of each other. Let
> me put it this way. Suppose I have 1000 insertions/seconds on a table,
> (each insert is independent & not coming in a list). Would it be a
> performance impact to use CP to update secondary index?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 2:20 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
> Deepak
> When you say 1000 invocations, you mean you are having a put(List)
> with 1000 items? In the region level when it deals with a batch of Puts,
> it calls postPut() for each of the Put one after the other. In the same
> handler thread which is doing the write. So yes one need to wait for the
> other. Otherwise you need to handle within the implementation of having a
> thread pool or so.
>
> I would say rethink on doing HTable#put() from a CP hook.
>
> -Anoop-
>
> ________________________________________
> From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> Sent: Monday, March 25, 2013 10:23 PM
> To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop
> Sam John
> Subject: RE: Regionserver goes down while endpoint execution
>
> Hi Anoop,
>
> Could you please provide some pointers. If we have a regionbase
> coprocessor which has some set of operation implemented in its postPut
> implementation.
>
> If I have say 1000 invocations of postPut per seconds, are those calls
> would be blocking? I mean, the second call will only start after the
> completion of first call of postPut, or it would execute independently?
>
> Regards,
> Deepak
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 8:59 AM
> To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
> What is this doing?
>
> HbaseTableAccess.getInstance().connectTable(tableName);
>
>
>
> Can you take a look at HTablePool?
>
>
>
> -Anoop-
>
>
>
> ________________________________
> From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 6:14 PM
> To: '[EMAIL PROTECTED]'; Anoop Sam John; '
> [EMAIL PROTECTED]'
> Subject: RE: Regionserver goes down while endpoint execution
>
>
> Hi,
>
> Even when I moved table > HbaseTableAccess.getInstance().connectTable(tableName); to postPut method
> of coprocessor, I am getting the same exception.
>
>
>
> Regards,
>
> Deepak
>
>
>
> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, March 20, 2013 4:37 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Regionserver goes down while endpoint execution
>
>
>
> Hi Deepak
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
-
Re: Regionserver goes down while endpoint executionAnoop John 2013-03-26, 17:17
I dont have any sample code to share with you. Instead of calling
HTable#put() from the postPut() impl, add these Put objects (Index table
Put) to some Q and the CP flow immediately comes out.(Not waiting for the
write on index table) Some dedicated pool of worker threads can poll the
index puts from Q and do the actual put() operation with index table. As
Adrien said this will be eventual consistent model and what if before the
worker thread actually doing the put on index table, the RS is going down.
So these consistency related points to be taken care. Hope I made it clear
to you :)
-Anoop-
On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Adrien,
> From CP hook, do we mean EndPoints implementation? Do we have any document
> for asynchronous put implementations? A simple example would also be quite
> better to start with.
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 3:42 AM
> To: [EMAIL PROTECTED]
> Cc: Anoop Sam John; [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> Sure, and it might be costly if your CP is building secondary index through
> `Put' to external tables/regions. Each of your RPC call will generate
> another RPC call. Also, take note that your client could get an unexpected
> behavior; it might be blocked waiting for RPC answer just because of your
> CP.
>
> As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> think about implementing asynchronous `Put' using a thread but it might
> lead in some headach (ensure - eventually - consistency, handle
> reconections, etc.)
>
>
> On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Thanks Anoop for assisting.
> >
> > I don't have Put(List). All those calls are independent of each other.
> Let
> > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > (each insert is independent & not coming in a list). Would it be a
> > performance impact to use CP to update secondary index?
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 2:20 AM
> > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > [EMAIL PROTECTED]'
> > Subject: RE: Regionserver goes down while endpoint execution
> >
> > Deepak
> > When you say 1000 invocations, you mean you are having a put(List)
> > with 1000 items? In the region level when it deals with a batch of Puts,
> > it calls postPut() for each of the Put one after the other. In the same
> > handler thread which is doing the write. So yes one need to wait for the
> > other. Otherwise you need to handle within the implementation of having a
> > thread pool or so.
> >
> > I would say rethink on doing HTable#put() from a CP hook.
> >
> > -Anoop-
> >
> > ________________________________________
> > From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> > Sent: Monday, March 25, 2013 10:23 PM
> > To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop
> > Sam John
> > Subject: RE: Regionserver goes down while endpoint execution
> >
> > Hi Anoop,
> >
> > Could you please provide some pointers. If we have a regionbase
> > coprocessor which has some set of operation implemented in its postPut
> > implementation.
> >
> > If I have say 1000 invocations of postPut per seconds, are those calls
> > would be blocking? I mean, the second call will only start after the
> > completion of first call of postPut, or it would execute independently?
> >
> > Regards,
> > Deepak
> > -----Original Message-----
> > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > Sent: Wednesday, March 20, 2013 8:59 AM
> > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > [EMAIL PROTECTED]'
> > Subject: RE: Regionserver goes down while endpoint execution
HTable#put() from the postPut() impl, add these Put objects (Index table
Put) to some Q and the CP flow immediately comes out.(Not waiting for the
write on index table) Some dedicated pool of worker threads can poll the
index puts from Q and do the actual put() operation with index table. As
Adrien said this will be eventual consistent model and what if before the
worker thread actually doing the put on index table, the RS is going down.
So these consistency related points to be taken care. Hope I made it clear
to you :)
-Anoop-
On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Adrien,
> From CP hook, do we mean EndPoints implementation? Do we have any document
> for asynchronous put implementations? A simple example would also be quite
> better to start with.
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 3:42 AM
> To: [EMAIL PROTECTED]
> Cc: Anoop Sam John; [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> Sure, and it might be costly if your CP is building secondary index through
> `Put' to external tables/regions. Each of your RPC call will generate
> another RPC call. Also, take note that your client could get an unexpected
> behavior; it might be blocked waiting for RPC answer just because of your
> CP.
>
> As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> think about implementing asynchronous `Put' using a thread but it might
> lead in some headach (ensure - eventually - consistency, handle
> reconections, etc.)
>
>
> On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Thanks Anoop for assisting.
> >
> > I don't have Put(List). All those calls are independent of each other.
> Let
> > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > (each insert is independent & not coming in a list). Would it be a
> > performance impact to use CP to update secondary index?
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 2:20 AM
> > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > [EMAIL PROTECTED]'
> > Subject: RE: Regionserver goes down while endpoint execution
> >
> > Deepak
> > When you say 1000 invocations, you mean you are having a put(List)
> > with 1000 items? In the region level when it deals with a batch of Puts,
> > it calls postPut() for each of the Put one after the other. In the same
> > handler thread which is doing the write. So yes one need to wait for the
> > other. Otherwise you need to handle within the implementation of having a
> > thread pool or so.
> >
> > I would say rethink on doing HTable#put() from a CP hook.
> >
> > -Anoop-
> >
> > ________________________________________
> > From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> > Sent: Monday, March 25, 2013 10:23 PM
> > To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop
> > Sam John
> > Subject: RE: Regionserver goes down while endpoint execution
> >
> > Hi Anoop,
> >
> > Could you please provide some pointers. If we have a regionbase
> > coprocessor which has some set of operation implemented in its postPut
> > implementation.
> >
> > If I have say 1000 invocations of postPut per seconds, are those calls
> > would be blocking? I mean, the second call will only start after the
> > completion of first call of postPut, or it would execute independently?
> >
> > Regards,
> > Deepak
> > -----Original Message-----
> > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > Sent: Wednesday, March 20, 2013 8:59 AM
> > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > [EMAIL PROTECTED]'
> > Subject: RE: Regionserver goes down while endpoint execution
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-28, 10:50
Hi Anoop,
Thanks for assisting. As Adrien said, it would be costly if we put to external tables/regions. Do we have a way so that we can force that the main table & secondary table is being loaded on same region server always, so that the Put is not as costly as in case of when they are distributed over network on different region servers?
Regards,
Deepak
-----Original Message-----
From: Anoop John [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 26, 2013 1:18 PM
To: [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
I dont have any sample code to share with you. Instead of calling
HTable#put() from the postPut() impl, add these Put objects (Index table
Put) to some Q and the CP flow immediately comes out.(Not waiting for the
write on index table) Some dedicated pool of worker threads can poll the
index puts from Q and do the actual put() operation with index table. As
Adrien said this will be eventual consistent model and what if before the
worker thread actually doing the put on index table, the RS is going down.
So these consistency related points to be taken care. Hope I made it clear
to you :)
-Anoop-
On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Adrien,
> From CP hook, do we mean EndPoints implementation? Do we have any document
> for asynchronous put implementations? A simple example would also be quite
> better to start with.
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 3:42 AM
> To: [EMAIL PROTECTED]
> Cc: Anoop Sam John; [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> Sure, and it might be costly if your CP is building secondary index through
> `Put' to external tables/regions. Each of your RPC call will generate
> another RPC call. Also, take note that your client could get an unexpected
> behavior; it might be blocked waiting for RPC answer just because of your
> CP.
>
> As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> think about implementing asynchronous `Put' using a thread but it might
> lead in some headach (ensure - eventually - consistency, handle
> reconections, etc.)
>
>
> On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Thanks Anoop for assisting.
> >
> > I don't have Put(List). All those calls are independent of each other.
> Let
> > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > (each insert is independent & not coming in a list). Would it be a
> > performance impact to use CP to update secondary index?
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 2:20 AM
> > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > [EMAIL PROTECTED]'
> > Subject: RE: Regionserver goes down while endpoint execution
> >
> > Deepak
> > When you say 1000 invocations, you mean you are having a put(List)
> > with 1000 items? In the region level when it deals with a batch of Puts,
> > it calls postPut() for each of the Put one after the other. In the same
> > handler thread which is doing the write. So yes one need to wait for the
> > other. Otherwise you need to handle within the implementation of having a
> > thread pool or so.
> >
> > I would say rethink on doing HTable#put() from a CP hook.
> >
> > -Anoop-
> >
> > ________________________________________
> > From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> > Sent: Monday, March 25, 2013 10:23 PM
> > To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop
> > Sam John
> > Subject: RE: Regionserver goes down while endpoint execution
> >
> > Hi Anoop,
> >
> > Could you please provide some pointers. If we have a regionbase
> > coprocessor which has some set of operation implemented in its postPut
> > implementation.
Thanks for assisting. As Adrien said, it would be costly if we put to external tables/regions. Do we have a way so that we can force that the main table & secondary table is being loaded on same region server always, so that the Put is not as costly as in case of when they are distributed over network on different region servers?
Regards,
Deepak
-----Original Message-----
From: Anoop John [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 26, 2013 1:18 PM
To: [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
I dont have any sample code to share with you. Instead of calling
HTable#put() from the postPut() impl, add these Put objects (Index table
Put) to some Q and the CP flow immediately comes out.(Not waiting for the
write on index table) Some dedicated pool of worker threads can poll the
index puts from Q and do the actual put() operation with index table. As
Adrien said this will be eventual consistent model and what if before the
worker thread actually doing the put on index table, the RS is going down.
So these consistency related points to be taken care. Hope I made it clear
to you :)
-Anoop-
On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Adrien,
> From CP hook, do we mean EndPoints implementation? Do we have any document
> for asynchronous put implementations? A simple example would also be quite
> better to start with.
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 3:42 AM
> To: [EMAIL PROTECTED]
> Cc: Anoop Sam John; [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> Sure, and it might be costly if your CP is building secondary index through
> `Put' to external tables/regions. Each of your RPC call will generate
> another RPC call. Also, take note that your client could get an unexpected
> behavior; it might be blocked waiting for RPC answer just because of your
> CP.
>
> As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> think about implementing asynchronous `Put' using a thread but it might
> lead in some headach (ensure - eventually - consistency, handle
> reconections, etc.)
>
>
> On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Thanks Anoop for assisting.
> >
> > I don't have Put(List). All those calls are independent of each other.
> Let
> > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > (each insert is independent & not coming in a list). Would it be a
> > performance impact to use CP to update secondary index?
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 2:20 AM
> > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > [EMAIL PROTECTED]'
> > Subject: RE: Regionserver goes down while endpoint execution
> >
> > Deepak
> > When you say 1000 invocations, you mean you are having a put(List)
> > with 1000 items? In the region level when it deals with a batch of Puts,
> > it calls postPut() for each of the Put one after the other. In the same
> > handler thread which is doing the write. So yes one need to wait for the
> > other. Otherwise you need to handle within the implementation of having a
> > thread pool or so.
> >
> > I would say rethink on doing HTable#put() from a CP hook.
> >
> > -Anoop-
> >
> > ________________________________________
> > From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> > Sent: Monday, March 25, 2013 10:23 PM
> > To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'; Anoop
> > Sam John
> > Subject: RE: Regionserver goes down while endpoint execution
> >
> > Hi Anoop,
> >
> > Could you please provide some pointers. If we have a regionbase
> > coprocessor which has some set of operation implemented in its postPut
> > implementation.
-
Re: Regionserver goes down while endpoint executionramkrishna vasudevan 2013-03-28, 10:53
Yes, you may have to write your custom load balancer that does that.
REgards
Ram
On Thu, Mar 28, 2013 at 4:20 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
> Thanks for assisting. As Adrien said, it would be costly if we put to
> external tables/regions. Do we have a way so that we can force that the
> main table & secondary table is being loaded on same region server always,
> so that the Put is not as costly as in case of when they are distributed
> over network on different region servers?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 1:18 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> I dont have any sample code to share with you. Instead of calling
> HTable#put() from the postPut() impl, add these Put objects (Index table
> Put) to some Q and the CP flow immediately comes out.(Not waiting for the
> write on index table) Some dedicated pool of worker threads can poll the
> index puts from Q and do the actual put() operation with index table. As
> Adrien said this will be eventual consistent model and what if before the
> worker thread actually doing the put on index table, the RS is going down.
> So these consistency related points to be taken care. Hope I made it clear
> to you :)
>
> -Anoop-
>
> On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Adrien,
> > From CP hook, do we mean EndPoints implementation? Do we have any
> document
> > for asynchronous put implementations? A simple example would also be
> quite
> > better to start with.
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 3:42 AM
> > To: [EMAIL PROTECTED]
> > Cc: Anoop Sam John; [EMAIL PROTECTED]
> > Subject: Re: Regionserver goes down while endpoint execution
> >
> > Sure, and it might be costly if your CP is building secondary index
> through
> > `Put' to external tables/regions. Each of your RPC call will generate
> > another RPC call. Also, take note that your client could get an
> unexpected
> > behavior; it might be blocked waiting for RPC answer just because of your
> > CP.
> >
> > As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> > think about implementing asynchronous `Put' using a thread but it might
> > lead in some headach (ensure - eventually - consistency, handle
> > reconections, etc.)
> >
> >
> > On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks Anoop for assisting.
> > >
> > > I don't have Put(List). All those calls are independent of each other.
> > Let
> > > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > > (each insert is independent & not coming in a list). Would it be a
> > > performance impact to use CP to update secondary index?
> > >
> > > Regards,
> > > Deepak
> > >
> > > -----Original Message-----
> > > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > > Sent: Tuesday, March 26, 2013 2:20 AM
> > > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > > [EMAIL PROTECTED]'
> > > Subject: RE: Regionserver goes down while endpoint execution
> > >
> > > Deepak
> > > When you say 1000 invocations, you mean you are having a put(List)
> > > with 1000 items? In the region level when it deals with a batch of
> Puts,
> > > it calls postPut() for each of the Put one after the other. In the same
> > > handler thread which is doing the write. So yes one need to wait for
> the
> > > other. Otherwise you need to handle within the implementation of
> having a
> > > thread pool or so.
> > >
> > > I would say rethink on doing HTable#put() from a CP hook.
> > >
> > > -Anoop-
> > >
> > > ________________________________________
> > > From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> > > Sent: Monday, March 25, 2013 10:23 PM
REgards
Ram
On Thu, Mar 28, 2013 at 4:20 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
> Thanks for assisting. As Adrien said, it would be costly if we put to
> external tables/regions. Do we have a way so that we can force that the
> main table & secondary table is being loaded on same region server always,
> so that the Put is not as costly as in case of when they are distributed
> over network on different region servers?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 1:18 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> I dont have any sample code to share with you. Instead of calling
> HTable#put() from the postPut() impl, add these Put objects (Index table
> Put) to some Q and the CP flow immediately comes out.(Not waiting for the
> write on index table) Some dedicated pool of worker threads can poll the
> index puts from Q and do the actual put() operation with index table. As
> Adrien said this will be eventual consistent model and what if before the
> worker thread actually doing the put on index table, the RS is going down.
> So these consistency related points to be taken care. Hope I made it clear
> to you :)
>
> -Anoop-
>
> On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Adrien,
> > From CP hook, do we mean EndPoints implementation? Do we have any
> document
> > for asynchronous put implementations? A simple example would also be
> quite
> > better to start with.
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 3:42 AM
> > To: [EMAIL PROTECTED]
> > Cc: Anoop Sam John; [EMAIL PROTECTED]
> > Subject: Re: Regionserver goes down while endpoint execution
> >
> > Sure, and it might be costly if your CP is building secondary index
> through
> > `Put' to external tables/regions. Each of your RPC call will generate
> > another RPC call. Also, take note that your client could get an
> unexpected
> > behavior; it might be blocked waiting for RPC answer just because of your
> > CP.
> >
> > As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> > think about implementing asynchronous `Put' using a thread but it might
> > lead in some headach (ensure - eventually - consistency, handle
> > reconections, etc.)
> >
> >
> > On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks Anoop for assisting.
> > >
> > > I don't have Put(List). All those calls are independent of each other.
> > Let
> > > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > > (each insert is independent & not coming in a list). Would it be a
> > > performance impact to use CP to update secondary index?
> > >
> > > Regards,
> > > Deepak
> > >
> > > -----Original Message-----
> > > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > > Sent: Tuesday, March 26, 2013 2:20 AM
> > > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > > [EMAIL PROTECTED]'
> > > Subject: RE: Regionserver goes down while endpoint execution
> > >
> > > Deepak
> > > When you say 1000 invocations, you mean you are having a put(List)
> > > with 1000 items? In the region level when it deals with a batch of
> Puts,
> > > it calls postPut() for each of the Put one after the other. In the same
> > > handler thread which is doing the write. So yes one need to wait for
> the
> > > other. Otherwise you need to handle within the implementation of
> having a
> > > thread pool or so.
> > >
> > > I would say rethink on doing HTable#put() from a CP hook.
> > >
> > > -Anoop-
> > >
> > > ________________________________________
> > > From: Kumar, Deepak8 [[EMAIL PROTECTED]]
> > > Sent: Monday, March 25, 2013 10:23 PM
-
RE: Regionserver goes down while endpoint executionKumar, Deepak8 2013-03-28, 12:11
Thanks Ram.
I am facing one more issue. I have hbase cluster which has 5 region servers & a master. I need to set one jar file on all regionservers classpath, so that it can be used by coprocessor. I stored the jar file in HDFS & trying to load it in the Distributed Cache(DistributedCache.addFileToClassPath(jmsPath, conf);)
But it is not being loaded. It is giving ClassNotFound exception. Once I copy it to all region servers lib location then it works. Does the Distributed cache not work for hbase?
Do we have any way so that the jar file is available to all region servers from a single location?
Regards,
Deepak
-----Original Message-----
From: ramkrishna vasudevan [mailto:[EMAIL PROTECTED]]
Sent: Thursday, March 28, 2013 6:54 AM
To: [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
Yes, you may have to write your custom load balancer that does that.
REgards
Ram
On Thu, Mar 28, 2013 at 4:20 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
> Thanks for assisting. As Adrien said, it would be costly if we put to
> external tables/regions. Do we have a way so that we can force that the
> main table & secondary table is being loaded on same region server always,
> so that the Put is not as costly as in case of when they are distributed
> over network on different region servers?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 1:18 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> I dont have any sample code to share with you. Instead of calling
> HTable#put() from the postPut() impl, add these Put objects (Index table
> Put) to some Q and the CP flow immediately comes out.(Not waiting for the
> write on index table) Some dedicated pool of worker threads can poll the
> index puts from Q and do the actual put() operation with index table. As
> Adrien said this will be eventual consistent model and what if before the
> worker thread actually doing the put on index table, the RS is going down.
> So these consistency related points to be taken care. Hope I made it clear
> to you :)
>
> -Anoop-
>
> On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Adrien,
> > From CP hook, do we mean EndPoints implementation? Do we have any
> document
> > for asynchronous put implementations? A simple example would also be
> quite
> > better to start with.
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 3:42 AM
> > To: [EMAIL PROTECTED]
> > Cc: Anoop Sam John; [EMAIL PROTECTED]
> > Subject: Re: Regionserver goes down while endpoint execution
> >
> > Sure, and it might be costly if your CP is building secondary index
> through
> > `Put' to external tables/regions. Each of your RPC call will generate
> > another RPC call. Also, take note that your client could get an
> unexpected
> > behavior; it might be blocked waiting for RPC answer just because of your
> > CP.
> >
> > As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> > think about implementing asynchronous `Put' using a thread but it might
> > lead in some headach (ensure - eventually - consistency, handle
> > reconections, etc.)
> >
> >
> > On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks Anoop for assisting.
> > >
> > > I don't have Put(List). All those calls are independent of each other.
> > Let
> > > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > > (each insert is independent & not coming in a list). Would it be a
> > > performance impact to use CP to update secondary index?
> > >
> > > Regards,
> > > Deepak
> > >
> > > -----Original Message-----
> > > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > > Sent: Tuesday, March 26, 2013 2:20 AM
I am facing one more issue. I have hbase cluster which has 5 region servers & a master. I need to set one jar file on all regionservers classpath, so that it can be used by coprocessor. I stored the jar file in HDFS & trying to load it in the Distributed Cache(DistributedCache.addFileToClassPath(jmsPath, conf);)
But it is not being loaded. It is giving ClassNotFound exception. Once I copy it to all region servers lib location then it works. Does the Distributed cache not work for hbase?
Do we have any way so that the jar file is available to all region servers from a single location?
Regards,
Deepak
-----Original Message-----
From: ramkrishna vasudevan [mailto:[EMAIL PROTECTED]]
Sent: Thursday, March 28, 2013 6:54 AM
To: [EMAIL PROTECTED]
Subject: Re: Regionserver goes down while endpoint execution
Yes, you may have to write your custom load balancer that does that.
REgards
Ram
On Thu, Mar 28, 2013 at 4:20 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
> Thanks for assisting. As Adrien said, it would be costly if we put to
> external tables/regions. Do we have a way so that we can force that the
> main table & secondary table is being loaded on same region server always,
> so that the Put is not as costly as in case of when they are distributed
> over network on different region servers?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 1:18 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> I dont have any sample code to share with you. Instead of calling
> HTable#put() from the postPut() impl, add these Put objects (Index table
> Put) to some Q and the CP flow immediately comes out.(Not waiting for the
> write on index table) Some dedicated pool of worker threads can poll the
> index puts from Q and do the actual put() operation with index table. As
> Adrien said this will be eventual consistent model and what if before the
> worker thread actually doing the put on index table, the RS is going down.
> So these consistency related points to be taken care. Hope I made it clear
> to you :)
>
> -Anoop-
>
> On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Adrien,
> > From CP hook, do we mean EndPoints implementation? Do we have any
> document
> > for asynchronous put implementations? A simple example would also be
> quite
> > better to start with.
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 3:42 AM
> > To: [EMAIL PROTECTED]
> > Cc: Anoop Sam John; [EMAIL PROTECTED]
> > Subject: Re: Regionserver goes down while endpoint execution
> >
> > Sure, and it might be costly if your CP is building secondary index
> through
> > `Put' to external tables/regions. Each of your RPC call will generate
> > another RPC call. Also, take note that your client could get an
> unexpected
> > behavior; it might be blocked waiting for RPC answer just because of your
> > CP.
> >
> > As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> > think about implementing asynchronous `Put' using a thread but it might
> > lead in some headach (ensure - eventually - consistency, handle
> > reconections, etc.)
> >
> >
> > On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks Anoop for assisting.
> > >
> > > I don't have Put(List). All those calls are independent of each other.
> > Let
> > > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > > (each insert is independent & not coming in a list). Would it be a
> > > performance impact to use CP to update secondary index?
> > >
> > > Regards,
> > > Deepak
> > >
> > > -----Original Message-----
> > > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > > Sent: Tuesday, March 26, 2013 2:20 AM
-
Re: Regionserver goes down while endpoint executionAgarwal, Saurabh 2013-03-28, 12:26
Thanks Ram.
Is there any blog or document regarding writing custom load balancer for secondary indexing? That will help tremendously.
Regards,
Saurabh.
----- Original Message -----
From: ramkrishna vasudevan [mailto:[EMAIL PROTECTED]]
Sent: Thursday, March 28, 2013 05:53 AM
To: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Yes, you may have to write your custom load balancer that does that.
REgards
Ram
On Thu, Mar 28, 2013 at 4:20 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
> Thanks for assisting. As Adrien said, it would be costly if we put to
> external tables/regions. Do we have a way so that we can force that the
> main table & secondary table is being loaded on same region server always,
> so that the Put is not as costly as in case of when they are distributed
> over network on different region servers?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 1:18 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> I dont have any sample code to share with you. Instead of calling
> HTable#put() from the postPut() impl, add these Put objects (Index table
> Put) to some Q and the CP flow immediately comes out.(Not waiting for the
> write on index table) Some dedicated pool of worker threads can poll the
> index puts from Q and do the actual put() operation with index table. As
> Adrien said this will be eventual consistent model and what if before the
> worker thread actually doing the put on index table, the RS is going down.
> So these consistency related points to be taken care. Hope I made it clear
> to you :)
>
> -Anoop-
>
> On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Adrien,
> > From CP hook, do we mean EndPoints implementation? Do we have any
> document
> > for asynchronous put implementations? A simple example would also be
> quite
> > better to start with.
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 3:42 AM
> > To: [EMAIL PROTECTED]
> > Cc: Anoop Sam John; [EMAIL PROTECTED]
> > Subject: Re: Regionserver goes down while endpoint execution
> >
> > Sure, and it might be costly if your CP is building secondary index
> through
> > `Put' to external tables/regions. Each of your RPC call will generate
> > another RPC call. Also, take note that your client could get an
> unexpected
> > behavior; it might be blocked waiting for RPC answer just because of your
> > CP.
> >
> > As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> > think about implementing asynchronous `Put' using a thread but it might
> > lead in some headach (ensure - eventually - consistency, handle
> > reconections, etc.)
> >
> >
> > On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks Anoop for assisting.
> > >
> > > I don't have Put(List). All those calls are independent of each other.
> > Let
> > > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > > (each insert is independent & not coming in a list). Would it be a
> > > performance impact to use CP to update secondary index?
> > >
> > > Regards,
> > > Deepak
> > >
> > > -----Original Message-----
> > > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > > Sent: Tuesday, March 26, 2013 2:20 AM
> > > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > > [EMAIL PROTECTED]'
> > > Subject: RE: Regionserver goes down while endpoint execution
> > >
> > > Deepak
> > > When you say 1000 invocations, you mean you are having a put(List)
> > > with 1000 items? In the region level when it deals with a batch of
> Puts,
> > > it calls postPut() for each of the Put one after the other. In the same
Is there any blog or document regarding writing custom load balancer for secondary indexing? That will help tremendously.
Regards,
Saurabh.
----- Original Message -----
From: ramkrishna vasudevan [mailto:[EMAIL PROTECTED]]
Sent: Thursday, March 28, 2013 05:53 AM
To: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Yes, you may have to write your custom load balancer that does that.
REgards
Ram
On Thu, Mar 28, 2013 at 4:20 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
> Thanks for assisting. As Adrien said, it would be costly if we put to
> external tables/regions. Do we have a way so that we can force that the
> main table & secondary table is being loaded on same region server always,
> so that the Put is not as costly as in case of when they are distributed
> over network on different region servers?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 1:18 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> I dont have any sample code to share with you. Instead of calling
> HTable#put() from the postPut() impl, add these Put objects (Index table
> Put) to some Q and the CP flow immediately comes out.(Not waiting for the
> write on index table) Some dedicated pool of worker threads can poll the
> index puts from Q and do the actual put() operation with index table. As
> Adrien said this will be eventual consistent model and what if before the
> worker thread actually doing the put on index table, the RS is going down.
> So these consistency related points to be taken care. Hope I made it clear
> to you :)
>
> -Anoop-
>
> On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Adrien,
> > From CP hook, do we mean EndPoints implementation? Do we have any
> document
> > for asynchronous put implementations? A simple example would also be
> quite
> > better to start with.
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 3:42 AM
> > To: [EMAIL PROTECTED]
> > Cc: Anoop Sam John; [EMAIL PROTECTED]
> > Subject: Re: Regionserver goes down while endpoint execution
> >
> > Sure, and it might be costly if your CP is building secondary index
> through
> > `Put' to external tables/regions. Each of your RPC call will generate
> > another RPC call. Also, take note that your client could get an
> unexpected
> > behavior; it might be blocked waiting for RPC answer just because of your
> > CP.
> >
> > As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> > think about implementing asynchronous `Put' using a thread but it might
> > lead in some headach (ensure - eventually - consistency, handle
> > reconections, etc.)
> >
> >
> > On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks Anoop for assisting.
> > >
> > > I don't have Put(List). All those calls are independent of each other.
> > Let
> > > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > > (each insert is independent & not coming in a list). Would it be a
> > > performance impact to use CP to update secondary index?
> > >
> > > Regards,
> > > Deepak
> > >
> > > -----Original Message-----
> > > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > > Sent: Tuesday, March 26, 2013 2:20 AM
> > > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > > [EMAIL PROTECTED]'
> > > Subject: RE: Regionserver goes down while endpoint execution
> > >
> > > Deepak
> > > When you say 1000 invocations, you mean you are having a put(List)
> > > with 1000 items? In the region level when it deals with a batch of
> Puts,
> > > it calls postPut() for each of the Put one after the other. In the same
-
RE: Regionserver goes down while endpoint executionAnoop Sam John 2013-04-02, 06:51
Saurabh,
Try reading this http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
-Anoop-
________________________________________
From: Agarwal, Saurabh [[EMAIL PROTECTED]]
Sent: Thursday, March 28, 2013 5:56 PM
To: '[EMAIL PROTECTED]'
Subject: Re: Regionserver goes down while endpoint execution
Thanks Ram.
Is there any blog or document regarding writing custom load balancer for secondary indexing? That will help tremendously.
Regards,
Saurabh.
----- Original Message -----
From: ramkrishna vasudevan [mailto:[EMAIL PROTECTED]]
Sent: Thursday, March 28, 2013 05:53 AM
To: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Yes, you may have to write your custom load balancer that does that.
REgards
Ram
On Thu, Mar 28, 2013 at 4:20 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
> Thanks for assisting. As Adrien said, it would be costly if we put to
> external tables/regions. Do we have a way so that we can force that the
> main table & secondary table is being loaded on same region server always,
> so that the Put is not as costly as in case of when they are distributed
> over network on different region servers?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 1:18 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> I dont have any sample code to share with you. Instead of calling
> HTable#put() from the postPut() impl, add these Put objects (Index table
> Put) to some Q and the CP flow immediately comes out.(Not waiting for the
> write on index table) Some dedicated pool of worker threads can poll the
> index puts from Q and do the actual put() operation with index table. As
> Adrien said this will be eventual consistent model and what if before the
> worker thread actually doing the put on index table, the RS is going down.
> So these consistency related points to be taken care. Hope I made it clear
> to you :)
>
> -Anoop-
>
> On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Adrien,
> > From CP hook, do we mean EndPoints implementation? Do we have any
> document
> > for asynchronous put implementations? A simple example would also be
> quite
> > better to start with.
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 3:42 AM
> > To: [EMAIL PROTECTED]
> > Cc: Anoop Sam John; [EMAIL PROTECTED]
> > Subject: Re: Regionserver goes down while endpoint execution
> >
> > Sure, and it might be costly if your CP is building secondary index
> through
> > `Put' to external tables/regions. Each of your RPC call will generate
> > another RPC call. Also, take note that your client could get an
> unexpected
> > behavior; it might be blocked waiting for RPC answer just because of your
> > CP.
> >
> > As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> > think about implementing asynchronous `Put' using a thread but it might
> > lead in some headach (ensure - eventually - consistency, handle
> > reconections, etc.)
> >
> >
> > On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks Anoop for assisting.
> > >
> > > I don't have Put(List). All those calls are independent of each other.
> > Let
> > > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > > (each insert is independent & not coming in a list). Would it be a
> > > performance impact to use CP to update secondary index?
> > >
> > > Regards,
> > > Deepak
> > >
> > > -----Original Message-----
> > > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > > Sent: Tuesday, March 26, 2013 2:20 AM
> > > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > > [EMAIL PROTECTED]'
Try reading this http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
-Anoop-
________________________________________
From: Agarwal, Saurabh [[EMAIL PROTECTED]]
Sent: Thursday, March 28, 2013 5:56 PM
To: '[EMAIL PROTECTED]'
Subject: Re: Regionserver goes down while endpoint execution
Thanks Ram.
Is there any blog or document regarding writing custom load balancer for secondary indexing? That will help tremendously.
Regards,
Saurabh.
----- Original Message -----
From: ramkrishna vasudevan [mailto:[EMAIL PROTECTED]]
Sent: Thursday, March 28, 2013 05:53 AM
To: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
Subject: Re: Regionserver goes down while endpoint execution
Yes, you may have to write your custom load balancer that does that.
REgards
Ram
On Thu, Mar 28, 2013 at 4:20 PM, Kumar, Deepak8 <[EMAIL PROTECTED]>wrote:
> Hi Anoop,
> Thanks for assisting. As Adrien said, it would be costly if we put to
> external tables/regions. Do we have a way so that we can force that the
> main table & secondary table is being loaded on same region server always,
> so that the Put is not as costly as in case of when they are distributed
> over network on different region servers?
>
> Regards,
> Deepak
>
> -----Original Message-----
> From: Anoop John [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 26, 2013 1:18 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Regionserver goes down while endpoint execution
>
> I dont have any sample code to share with you. Instead of calling
> HTable#put() from the postPut() impl, add these Put objects (Index table
> Put) to some Q and the CP flow immediately comes out.(Not waiting for the
> write on index table) Some dedicated pool of worker threads can poll the
> index puts from Q and do the actual put() operation with index table. As
> Adrien said this will be eventual consistent model and what if before the
> worker thread actually doing the put on index table, the RS is going down.
> So these consistency related points to be taken care. Hope I made it clear
> to you :)
>
> -Anoop-
>
> On Tue, Mar 26, 2013 at 1:57 PM, Kumar, Deepak8 <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Adrien,
> > From CP hook, do we mean EndPoints implementation? Do we have any
> document
> > for asynchronous put implementations? A simple example would also be
> quite
> > better to start with.
> >
> > Regards,
> > Deepak
> >
> > -----Original Message-----
> > From: Adrien Mogenet [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, March 26, 2013 3:42 AM
> > To: [EMAIL PROTECTED]
> > Cc: Anoop Sam John; [EMAIL PROTECTED]
> > Subject: Re: Regionserver goes down while endpoint execution
> >
> > Sure, and it might be costly if your CP is building secondary index
> through
> > `Put' to external tables/regions. Each of your RPC call will generate
> > another RPC call. Also, take note that your client could get an
> unexpected
> > behavior; it might be blocked waiting for RPC answer just because of your
> > CP.
> >
> > As Anoop said, rethink on doing such things from your CP hook. Otherwise,
> > think about implementing asynchronous `Put' using a thread but it might
> > lead in some headach (ensure - eventually - consistency, handle
> > reconections, etc.)
> >
> >
> > On Tue, Mar 26, 2013 at 8:27 AM, Kumar, Deepak8 <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks Anoop for assisting.
> > >
> > > I don't have Put(List). All those calls are independent of each other.
> > Let
> > > me put it this way. Suppose I have 1000 insertions/seconds on a table,
> > > (each insert is independent & not coming in a list). Would it be a
> > > performance impact to use CP to update secondary index?
> > >
> > > Regards,
> > > Deepak
> > >
> > > -----Original Message-----
> > > From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> > > Sent: Tuesday, March 26, 2013 2:20 AM
> > > To: Kumar, Deepak8 [CCC-OT_IT NE]; '[EMAIL PROTECTED]'; '
> > > [EMAIL PROTECTED]'
