Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> SequenceFileLogReader uses a reflection hack resulting in runtime failures


Copy link to this message
-
SequenceFileLogReader uses a reflection hack resulting in runtime failures
Hello,

The following reflection hack is from SequenceFileLogReader.java.

         try {
           Field fIn = FilterInputStream.class.getDeclaredField("in");
           fIn.setAccessible(true);
           Object realIn = fIn.get(this.in);
           Method getFileLength = realIn.getClass().
             getMethod("getFileLength", new Class<?> []{});
           getFileLength.setAccessible(true);
           long realLength = ((Long)getFileLength.
             invoke(realIn, new Object []{})).longValue();
           assert(realLength >= this.length);
           adjust = realLength - this.length;
         } catch(Exception e) {
           SequenceFileLogReader.LOG.warn(
             "Error while trying to get accurate file length.  " +
             "Truncation / data loss may occur if RegionServers die.", e);
         }

In my testing this resulted in the following error:

11/12/01 21:40:07 WARN wal.SequenceFileLogReader: Error while trying to get
accurate file length.  Truncation / data loss may occur if RegionServers
die.
java.lang.NoSuchMethodException:
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.getFileLength()
        at java.lang.Class.getMethod(Class.java:1605)
        at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader$WALReaderFSDataInputStream.getPos(SequenceFileLogReader.java:108)
        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1434)
        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419)
        at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:57)
        at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:158)
        at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:681)
        at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:821)
        at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:734)
        at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFileToTemp(HLogSplitter.java:382)
        at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFileToTemp(HLogSplitter.java:351)
        at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:113)
        at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:266)
        at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:197)
        at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:165)
        at java.lang.Thread.run(Thread.java:680)

Besides, even when it works, the reflection-based solution is probably
_much_ slower than straightforward method access. Should we create a Hadoop
patch to expose the appropriate API call and get rid of the reflection hack?

Thanks,
--Mikhail
+
Dhruba Borthakur 2011-12-02, 06:32
+
Stack 2011-12-02, 06:33
+
Mikhail Bautin 2011-12-02, 06:58
+
Mikhail Bautin 2011-12-02, 07:12
+
Dhruba Borthakur 2011-12-02, 07:16
+
Mikhail Bautin 2011-12-02, 07:20
+
Mikhail Bautin 2011-12-02, 08:54
+
Todd Lipcon 2011-12-02, 07:18
+
Stack 2011-12-02, 14:55
+
Nicolas Spiegelberg 2011-12-02, 16:39
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB