Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Cluster crash


Copy link to this message
-
Re: Cluster crash
This is how it's configured in /etc/security/limits.con on all the
slaves in the cluster:
hadoop           -       nofile          32768
hdfs             -       nofile          32768
hbase            -       nofile          32768
hadoop           -       nproc           32000
hdfs           -       nproc           32000
hbase           -       nproc           32000

When hbase is loading it prints:
ulimit -n 32768
-eran
On Sun, Apr 10, 2011 at 21:54, Stack <[EMAIL PROTECTED]> wrote:
> Did you read the requirements section [1] and verify that indeed
> ulimit and nprocs for the user who owns hbase and hadoop processes has
> indeed the upped limits?
>
> Yours,
> St.Ack
>
> 1. http://hbase.apache.org/book/notsoquick.html#requirements
>
> On Sun, Apr 10, 2011 at 8:07 AM, Eran Kutner <[EMAIL PROTECTED]> wrote:
>> Hi,
>> While doing load testing on HBase the entire cluster crashed with
>> errors like these in hbase logs:
>>
>> 2011-04-10 10:14:30,844 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block blk_1213779416283711358_54194 bad datanode[0]
>> 10.1.104.1:50010
>> 2011-04-10 10:14:30,844 WARN org.apache.hadoop.hdfs.DFSClient: Error
>> Recovery for block blk_1213779416283711358_54194 in pipeline
>> 10.1.104.1:50010, 10.1.104.5:50010, 10.1.104.2:50010: bad datanode
>> 10.1.104.1:50010
>> 2011-04-10 10:14:30,880 WARN org.apache.hadoop.hdfs.DFSClient: Failed
>> recovery attempt #2 from primary datanode 10.1.104.2:50010
>> org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Block
>> (=blk_1213779416283711358_54194) not found
>>        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchronization(FSNamesystem.java:2099)
>>        at org.apache.hadoop.hdfs.server.namenode.NameNode.commitBlockSynchronization(NameNode.java:703)
>>        at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1416)
>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1412)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at javax.security.auth.Subject.doAs(Subject.java:396)
>>        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1410)
>>
>>        at org.apache.hadoop.ipc.Client.call(Client.java:1104)
>>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
>>        at $Proxy4.commitBlockSynchronization(Unknown Source)
>>        at org.apache.hadoop.hdfs.server.datanode.DataNode.syncBlock(DataNode.java:1847)
>>        at org.apache.hadoop.hdfs.server.datanode.DataNode.recoverBlock(DataNode.java:1828)
>>        at org.apache.hadoop.hdfs.server.datanode.DataNode.recoverBlock(DataNode.java:1924)
>>        at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1416)
>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1412)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at javax.security.auth.Subject.doAs(Subject.java:396)
>>        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1410)
>>
>>        at org.apache.hadoop.ipc.Client.call(Client.java:1104)
>>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
>>        at $Proxy8.recoverBlock(Unknown Source)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB