Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Re: org.apache.hadoop.hbase.ZooKeeperConnectionException: An error is preventing HBase from connecting to ZooKeeper


Copy link to this message
-
Re: org.apache.hadoop.hbase.ZooKeeperConnectionException: java.io.IOException: Too many open files
Ian Varley 2011-08-09, 15:38
Hi Shuja,

This question is mentioned in the HBase FAQ, here:

http://wiki.apache.org/hadoop/Hbase/FAQ_Operations#A3

which points to the HBase book:

http://hbase.apache.org/book.html#ulimit

"HBase is a database. It uses a lot of files all at the same time. The default ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems is insufficient (On mac os x its 256). Any significant amount of loading will lead you to "FAQ: Why do I see "java.io.IOException...(Too many open files)" in my logs?" ... Do yourself a favor and change the upper bound on the number of file descriptors. Set it to north of 10k. See the above referenced FAQ for how. You should also up the hbase users' nproc setting; under load, a low-nproc setting could manifest as OutOfMemoryError. To be clear, upping the file descriptors and nproc for the user who is running the HBase process is an operating system configuration, not an HBase configuration. Also, a common mistake is that administrators will up the file descriptors for a particular user but for whatever reason, HBase will be running as some one else. HBase prints in its logs as the first line the ulimit its seeing. Ensure its correct."

- Ian

On Aug 9, 2011, at 10:32 AM, Shuja Rehman wrote:

Hi All,

I am running map reduce jobs in queue and after some jobs, the following
exception starts to come. Let me know if I need to change any settings or
anything else

Thanks in advance.
11/08/09 06:19:56 INFO mapreduce.TableOutputFormat: Created table instance
for mytable
11/08/09 06:19:56 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=10.0.3.85:2181 sessionTimeout=180000 watcher=hconnection
11/08/09 06:19:56 ERROR mapreduce.TableInputFormat:
org.apache.hadoop.hbase.ZooKeeperConnectionException: java.io.IOException:
Too many open files
       at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
       at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
       at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
       at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
       at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
       at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
       at
org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:91)
       at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
       at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
       at
org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:941)
       at
org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)
       at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
       at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
       at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:396)
       at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
       at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
       at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
       at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
       at
hbaseaggregator.AggregatorRunner.runJob(AggregatorRunner.java:242)
       at
hbaseaggregator.AggregatorRunner.Aggregator(AggregatorRunner.java:180)
       at hbaseaggregator.AggregatorDriver.main(AggregatorDriver.java:60)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: java.io.IOException: Too many open files
       at sun.nio.ch.IOUtil.initPipe(Native Method)
       at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
       at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
       at java.nio.channels.Selector.open(Selector.java:209)
       at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:160)
       at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:331)
       at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:377)
       at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:97)
       at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:119)
       at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
       ... 27 more

11/08/09 06:19:56 INFO mapred.JobClient: Cleaning up the staging area
hdfs://
hadoop.zoniversal.com/var/lib/hadoop-0.20/cache/mapred/mapred/staging/root/.staging/job_201107251124_0383<http://hadoop.zoniversal.com/var/lib/hadoop-0.20/cache/mapred/mapred/staging/root/.staging/job_201107251124_0383>
Exception in thread "main" java.lang.InternalError
       at
sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:755)
       at sun.misc.URLClassPath.getResource(URLClassPath.java:169)
       at java.net.URLClassLoader$1.run(URLClassLoader.java:194)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
       at sun.misc.Launcher$ExtClassLoader.findClass(Launcher.java:229)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at java.lang.ClassLo