Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Some questions on HLog


Copy link to this message
-
Re: Some questions on HLog
This was due to the fact that when HBASE-2312 was integrated, there were
many flavors of hadoop running in production.
So the code had to support all the flavors.

Cheers
On Thu, Oct 24, 2013 at 9:27 AM, Wukang Lin <[EMAIL PROTECTED]> wrote:

> Hi Ted,
>     Thank you for your response. for #1, I have tried to understand that
> comment in SequenceFileLogWriter, i can't figure out that instead of
> reflection, why not use the version of SF.createWriter below directly?
>     SequenceFile.Writer createWriter(FileSystem fs,
>                                                      Configuration conf,
>                                                      Path name,
>                                                      Class keyClass,
>                                                      Class valClass,
>                                                      int bufferSize,
>                                                      short replication,
>                                                      long blockSize,
>                                                      boolean createParent,
>                                                      CompressionType
> compressionType,
>                                                      CompressionCodec
> codec,
>                                                      Metadata metadata)
> throws java.io.IOException;
>
>     Thank you again.
>
>
>
> 2013/10/24 Ted Yu <[EMAIL PROTECTED]>
>
> > For #2, see HBASE-5954
> >
> > For #1, see the following comment in SequenceFileLogWriter:
> >
> > +      // reflection for a version of SequenceFile.createWriter that
> > doesn't
> > +      // automatically create the parent directory (see HBASE-2312)
> > +      this.writer = (SequenceFile.Writer) SequenceFile.class
> >
> >
> >
> > On Thu, Oct 24, 2013 at 8:49 AM, Wukang Lin <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hi all,
> > >     Recently, i read the source of HBase's HLog, and i got some
> questions
> > > that puzzled me a lot. here there are:
> > >     1 why use reflection to init a SequenceFile.Writer
> > > in SequenceFileLogWriter? i read HBASE-2312 but still can't catch the
> > > point.
> > >      2 It seems that hlog use SequenceFile.Writer's append method to
> sync
> > > the WAL logs to DataNode, not FSDataOutputStream.hflush(), for each
> > > mutation(or batch mutations), so may it lose data when HDFS crash while
> > WAL
> > > logs were 'sync' to DataNode but not flush to disk? or are there
> > something
> > > i misunderstanded?
> > >
> > >       Thank you.
> > >
> >
>