Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> HLogSplit error with hadoop-2.0.3-alpha and hbase trunk


Copy link to this message
-
Re: HLogSplit error with hadoop-2.0.3-alpha and hbase trunk
I see we have FSDataInputStream in ProtobufLogReader.. We will test with
the patch tomorow and report back.

-Anoop-

On Thu, May 9, 2013 at 12:30 AM, Anoop John <[EMAIL PROTECTED]> wrote:

>  >However this doesn't explain why sequencefile has no such problems
>
> What I can see is we use FSDataInputStream in case of
> SequenecFileLogReader (Not FSInputStream) and we use FSDataInputStream
> #readInt() (WALEdit#read...).  This inturn reads the int from stream by
> calling in.read() 4 times making sure we read full int (If it is not EOF
> issue)
>
> May be we can try out this way and test tomorow Ram.
>
> BTW IOUtils.readFully() is not returning the actual # bytes read it looks.
>
> -Anoop-
>
>  On Thu, May 9, 2013 at 12:07 AM, Sergey Shelukhin <[EMAIL PROTECTED]
> > wrote:
>
>> 589824 is divisible by 4096, default buffer size. Que groundless
>> suspicions. However this doesn't explain why sequencefile has no such
>> problems. Do you have any more numbers with this?
>> I am in meetings now, let me try readFully after I come back from lunch
>>
>> On Wed, May 8, 2013 at 11:23 AM, ramkrishna vasudevan <
>>  [EMAIL PROTECTED]> wrote:
>>
>> > >The length is from read call. This looks pretty suspicious, if the
>> stream
>> > is not EOF why would it return less bytes?
>> >
>> > Infact this happened for an entry in the middle of the HLog.  Write some
>> > data using LoadTestTool.  Try to use the HLog tool to read the HLog file
>> > you will end up in this issue. (most of the times).
>> >
>> > Regards
>> > Ram
>> >
>> >
>> >
>> > On Wed, May 8, 2013 at 11:48 PM, Sergey Shelukhin <
>> [EMAIL PROTECTED]
>> > >wrote:
>> >
>> > > if (length != intBytes.length) throw new IOException("Failed read of
>> int
>> > > length " + length);
>> > > The length is from read call. This looks pretty suspicious, if the
>> stream
>> > > is not EOF why would it return less bytes? I will try to repro today.
>> > >
>> > > On Wed, May 8, 2013 at 5:46 AM, ramkrishna vasudevan <
>> > > [EMAIL PROTECTED]> wrote:
>> > >
>> > > > On further debugging found that this issue happens with
>> ProtoBufWriter
>> > > and
>> > > > not with sequenceFileWriter.(atleast we could not reproduce it with
>> > > > different runs)
>> > > >
>> > > > We can see that the HLog has more data in it but while reading one
>> of
>> > the
>> > > > lines in the HLog this error happens.  So pretty much sure that it
>> is
>> > not
>> > > > EOF.
>> > > > Verified DFS logs but could not find any exceptions out there too.
>> > > >
>> > > > We will try to figure out more on this tomorrow.
>> > > >
>> > > > Regards
>> > > > Ram
>> > > >
>> > > >
>> > > > On Wed, May 8, 2013 at 11:34 AM, ramkrishna vasudevan <
>> > > > [EMAIL PROTECTED]> wrote:
>> > > >
>> > > > > Ok so i tried this out with hadoop 2.0.4 and also with Sergey's
>> > patch.
>> > > > >  The issue is reproducible in all version of hadoop but not
>> always.
>> > > > > I am able to get the errors like this
>> > > > >
>> > > > > 2013-05-07 17:11:08,827 WARN  [SplitLogWorker-ram.sh.intel.com<http://splitlogworker-ram.sh.intel.com/>
>> > > > ,60020,1367961009182]
>> > > > > org.apache.hadoop.hbase.regionserver.SplitLogWorker: log
>> splitting of
>> > > > .logs/
>> > > > > ram.sh.intel.com,60020,1367960957620-splitting/ram.sh.intel.com
>> > > > %2C60020%2C1367960957620.1367960993389
>> > > > > failed, returning error
>> > > > > java.io.IOException: Error  while reading 1 WAL KVs; started
>> reading
>> > at
>> > > > > 589822 and read up to 589824
>> > > > > at
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:162)
>> > > > >  at
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:88)
>> > > > > at
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:75)
>> > > > >  at
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getNextLogLine(HLogSplitter.java:775)