Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> HLogSplit error with hadoop-2.0.3-alpha and hbase trunk


+
ramkrishna vasudevan 2013-05-06, 11:26
+
Sergey Shelukhin 2013-05-06, 18:36
+
Sergey Shelukhin 2013-05-06, 23:11
+
ramkrishna vasudevan 2013-05-07, 02:42
+
ramkrishna vasudevan 2013-05-08, 06:04
+
ramkrishna vasudevan 2013-05-08, 12:46
+
Sergey Shelukhin 2013-05-08, 18:18
+
ramkrishna vasudevan 2013-05-08, 18:23
+
Sergey Shelukhin 2013-05-08, 18:37
+
Anoop John 2013-05-08, 19:00
+
Anoop John 2013-05-08, 19:17
+
Jimmy Xiang 2013-05-08, 19:46
+
Anoop John 2013-05-08, 18:34
Copy link to this message
-
Re: HLogSplit error with hadoop-2.0.3-alpha and hbase trunk
Ram,
  For you the issue was not happening with lesser data right?

-Anoop-

On Thu, May 9, 2013 at 12:04 AM, Anoop John <[EMAIL PROTECTED]> wrote:

> The issue was not coming for the same test with same ammount of data with
> SequenceFileLogWritter & SFLR
>
> Can the FSInputStream#read(byte[]) can read lesser bytes only even when it
> is not EOF?  I can see we use   IOUtils.*readFully*(in, byte[], int, int);
> What is the difference of this from the other?  Do there be a diff when we
> read at file blocks boundary? (For these 2)
>
> -Anoop-
>
>  On Wed, May 8, 2013 at 11:53 PM, ramkrishna vasudevan <
> [EMAIL PROTECTED]> wrote:
>
>> >The length is from read call. This looks pretty suspicious, if the stream
>> is not EOF why would it return less bytes?
>>
>> Infact this happened for an entry in the middle of the HLog.  Write some
>> data using LoadTestTool.  Try to use the HLog tool to read the HLog file
>> you will end up in this issue. (most of the times).
>>
>> Regards
>> Ram
>>
>>
>>
>> On Wed, May 8, 2013 at 11:48 PM, Sergey Shelukhin <[EMAIL PROTECTED]
>> >wrote:
>>
>> > if (length != intBytes.length) throw new IOException("Failed read of int
>> > length " + length);
>> > The length is from read call. This looks pretty suspicious, if the
>> stream
>> > is not EOF why would it return less bytes? I will try to repro today.
>> >
>> > On Wed, May 8, 2013 at 5:46 AM, ramkrishna vasudevan <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> > > On further debugging found that this issue happens with ProtoBufWriter
>> > and
>> > > not with sequenceFileWriter.(atleast we could not reproduce it with
>> > > different runs)
>> > >
>> > > We can see that the HLog has more data in it but while reading one of
>> the
>> > > lines in the HLog this error happens.  So pretty much sure that it is
>> not
>> > > EOF.
>> > > Verified DFS logs but could not find any exceptions out there too.
>> > >
>> > > We will try to figure out more on this tomorrow.
>> > >
>> > > Regards
>> > > Ram
>> > >
>> > >
>> > > On Wed, May 8, 2013 at 11:34 AM, ramkrishna vasudevan <
>> > > [EMAIL PROTECTED]> wrote:
>> > >
>> > > > Ok so i tried this out with hadoop 2.0.4 and also with Sergey's
>> patch.
>> > > >  The issue is reproducible in all version of hadoop but not always.
>> > > > I am able to get the errors like this
>> > > >
>> > > > 2013-05-07 17:11:08,827 WARN  [SplitLogWorker-ram.sh.intel.com<http://splitlogworker-ram.sh.intel.com/>
>> > > ,60020,1367961009182]
>> > > > org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting
>> of
>> > > .logs/
>> > > > ram.sh.intel.com,60020,1367960957620-splitting/ram.sh.intel.com
>> > > %2C60020%2C1367960957620.1367960993389
>> > > > failed, returning error
>> > > > java.io.IOException: Error  while reading 1 WAL KVs; started
>> reading at
>> > > > 589822 and read up to 589824
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:162)
>> > > >  at
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:88)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:75)
>> > > >  at
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getNextLogLine(HLogSplitter.java:775)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:459)
>> > > >  at
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:388)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
>> > > >  at
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:278)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:199)
>>
+
Jean-Marc Spaggiari 2013-05-06, 14:55
+
Ted Yu 2013-05-06, 12:21
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB