Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Region Servers crashing following: "File does not exist", "Too many open files" exceptions


Copy link to this message
-
Re: : Region Servers crashing following: "File does not exist", "Too many open files" exceptions
Hello,

Thank you for your replies. In the end we dropped the concerned tables and
are in the process of re-importing data. Looking through the mailing list
it seems like this issue [1] may be identical to what we are experiencing.
TLDR: Region splits fail when there is a lack of disk space, leaving some
orphan references to non-existant regions which HBase tries to access
viciously, exhausting file handlers in the process thereby degrading Region
Server performance. There is a JIRA for this [2].

We looked for references to said files and deleted them but we must have
missed something because hbase hbck -repair still stalls. In any case, our
bad for letting the cluster get to the point where there was hardly any
disk space.

If someone reading this has experienced the same problem but managed to
restore order without resorting to drastic measures such as dropping a
table I'd be curious to know about the steps that were taken.

Thank you,

/David
[1]
http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCAO=qdPQ1jJaaXCt2CVpHZev7q-QHR1x4D+[EMAIL PROTECTED]%3E
[2] https://issues.apache.org/jira/browse/HBASE-7335

On Mon, Feb 11, 2013 at 5:50 PM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> From the UI can you figure out how many store files are present?  Also if
> you can check the logs it will tel you if the compactions were happening.
> I may be wrong without checking your cluster, just some inputs that we have
> faced sometime back.
>
> Regards
> Ram
>
> On Mon, Feb 11, 2013 at 8:54 PM, David Koch <[EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > No, we did not change anything, so compactions should run at
> automatically
> > - I guess it's once a day - however, I don't know to what extent jobs
> > running on the cluster have impeded compactions - if this is even a
> > possibility.
> >
> > /David
> >
> > On Mon, Feb 11, 2013 at 4:58 AM, ramkrishna vasudevan <
> > [EMAIL PROTECTED]> wrote:
> >
> > > Hi David,
> > >
> > > Have you changed anything on the configurations related to compactions?
> > >
> > > If there are more store files created and if the compactions are not
> run
> > > frequently we end up in this problem.  Atleast there will be a
> consistent
> > > increase in the file handler count.
> > >
> > > Could you run compactions manually to see if it helps?
> > >
> > > Regards
> > > Ram
> > >
> > > On Mon, Feb 11, 2013 at 1:41 AM, David Koch <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Like I said, the maximum permissible number of filehandlers is set to
> > > 65535
> > > > for users hbase (the one who starts HBase), mapred and hdfs
> > > >
> > > > The too many files warning occurs on the region servers but not on
> the
> > > HDFS
> > > > namenode.
> > > >
> > > > /David
> > > >
> > > >
> > > > On Sun, Feb 10, 2013 at 3:53 PM, shashwat shriparv <
> > > > [EMAIL PROTECTED]> wrote:
> > > >
> > > > > On Sun, Feb 10, 2013 at 6:21 PM, David Koch <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > >
> > > > > > problems but could not find any. The settings
> > > > >
> > > > >
> > > > > increase the u limit for the user using you are starting the hadoop
> > and
> > > > > hbase services, in os
> > > > >
> > > > >
> > > > >
> > > > > ∞
> > > > > Shashwat Shriparv
> > > > >
> > > >
> > >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB