Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - HBase Checksum


Copy link to this message
-
Re: HBase Checksum
Robert Dyer 2013-02-01, 06:23
Ok grepping the RS logs I see nothing with 'local' in any of them.  Thanks
for that hint.

For the test I was using, I know it is data local.  Every map task launched
data local, and no regions were moving recently.

I think I've hijacked this thread enough, I'll move my issues to another.
;-)
On Thu, Jan 31, 2013 at 11:51 PM, Anoop Sam John <[EMAIL PROTECTED]> wrote:

> Hi Robert
>           When HDFS is doing the local short circuit read, it will use
> BlockReaderLocal class for reading.  There should be some logs at the DFS
> client side (RS) which tells abt creating new BlockReaderLocal .  If you
> can see this then sure the local read is happening.
>
> Also check DN log.  If local read happening, then you will not see  read
> request related logs for the HFile at the DN side.
> You check your no# of HFiles and names for checking the logs
>
> Are you sure that when you tested, u have data locality? Region movements
> across RSs can break the full data locality.
>
> -Anoop-
> ________________________________________
> From: Robert Dyer [[EMAIL PROTECTED]]
> Sent: Friday, February 01, 2013 11:10 AM
> To: Hbase-User
> Subject: Re: HBase Checksum
>
> Not trying to hijack your thread here...
>
> But can you verify via logs that the shortcircuit is working?  Because I
> enabled shortcircuit but I sure didn't see any performance increase.
>
> I haven't tried enabling hbase checksum yet but I'd like to be able to
> verify that works too.
>
>
> On Thu, Jan 31, 2013 at 9:55 PM, Anoop Sam John <[EMAIL PROTECTED]>
> wrote:
>
> > You can check with HDFS level logs whether the checksum meta file is
> > getting read to the DFS client? In the HBase handled checksum, this
> should
> > not happen.
> > Have you noticed any perf gain when you configure the HBase handled
> > checksum option?
> >
> > -Anoop-
> > ________________________________________
> > From: Jean-Marc Spaggiari [[EMAIL PROTECTED]]
> > Sent: Friday, February 01, 2013 4:16 AM
> > To: user
> > Subject: HBase Checksum
> >
> > Hi,
> >
> > I have activated shortcircuit and checksum and I would like to get a
> > confirmation that it's working fine.
> >
> > So I have activated short circuit first and saw a 40% improvement of
> > the MR rowcount job. So I guess it's working fine.
> >
> > Now, I'm configuring the checksum option, and I'm wondering how I can
> > do to validate that it's taken into consideration and used, or not. Is
> > there a way to see that?
> >
> > Thanks,
> >
> > JM
> >
>

--

Robert Dyer
[EMAIL PROTECTED]