Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Replication warning


+
Álvaro Recuero 2013-05-27, 12:15
+
Ted Yu 2013-05-27, 13:43
+
Álvaro Recuero 2013-05-27, 15:40
Copy link to this message
-
Re: Replication warning
There was a recently integrated JIRA (0.94.8):
HBASE-7122 Proper warning message when opening a log file with no entries
(idle cluster)

The stack trace in above JIRA looks very similar to yours.

On Mon, May 27, 2013 at 8:40 AM, Álvaro Recuero <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> From the shell:
>
> ~/hbase-0.94.4$ ./bin/hbase shell
> HBase Shell; enter 'help<RETURN>' for list of supported commands.
> Type "exit<RETURN>" to leave the HBase Shell
> Version 0.94.4, r, Mon May 27 15:29:11 WEST 2013
>
> I have seen today it comes up as soon as there is a new item to be
> replicated, and even happens if an old Hlog is rolled. I am not really sure
> what is going there behind the scenes of Hbase.
>
> Regards,
> Álvaro
>
> On 27 May 2013 14:43, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > The warning came from this loop in ReplicationSource:
> >
> >     // Loop until we close down
> >     while (isActive()) {
> > ...
> >       // Open a reader on it
> >       if (!openReader(sleepMultiplier)) {
> >         // Reset the sleep multiplier, else it'd be reused for the next
> > file
> >         sleepMultiplier = 1;
> >         continue;
> >       }
> >
> >       // If we got a null reader but didn't continue, then sleep and
> > continue
> >       if (this.reader == null) {
> >         if (sleepForRetries("Unable to open a reader", sleepMultiplier))
> {
> >           sleepMultiplier++;
> >         }
> >         continue;
> >       }
> >
> > How often did the following warning come up in the server log ?
> > Which version of HBase are you using ?
> >
> > Cheers
> >
> > On Mon, May 27, 2013 at 5:15 AM, Álvaro Recuero <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Does anyone get this when replication is enabled in HBase? If it is a
> > > warning but does not stop coming up, I wonder what that loop is that
> > > iterating through and if this is really a critical problem.
> > >
> > > cheers,
> > > Álvaro
> > >
> > > 2013-05-27 13:10:40,406 WARN
> > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > Waited
> > > too long for this file, considering dumping
> > > 2013-05-27 13:10:40,406 DEBUG
> > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> > Unable
> > > to open a reader, sleeping 1000 times 10
> > > 2013-05-27 13:10:50,411 WARN
> > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1
> > Got:
> > > java.io.EOFException
> > > at java.io.DataInputStream.readFully(DataInputStream.java:197)
> > >  at java.io.DataInputStream.readFully(DataInputStream.java:169)
> > > at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
> > >  at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
> > > at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> > >  at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
> > >  at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
> > > at
> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:721)
> > >  at
> > >
> > >
> >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:562)
> > >  at
> > >
> > >
> >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:339)
> > >
> >
>
+
Álvaro Recuero 2013-05-28, 12:23
+
Ted Yu 2013-05-28, 12:41
+
Álvaro Recuero 2013-05-30, 10:27
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB