Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Wrong input split locations after enabling reverse DNS


Copy link to this message
-
Re: Wrong input split locations after enabling reverse DNS
New issue, the other one is too old.

Thx!

J-D

On Mon, Dec 17, 2012 at 6:39 PM, Robert Dyer <[EMAIL PROTECTED]> wrote:
> Seems plausible.  A simple grep reveals this:
>
> mapreduce/TableInputFormatBase.java:      hostName > DNS.reverseDns(ipAddress, this.nameServer);
>
> which is not doing the filtering that HBASE-4109 does.
>
> Would this typically be filed as a new issue or brought up in comments on
> the closed issue?
>
>
> On Mon, Dec 17, 2012 at 8:21 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>
> wrote:
>>
>> Maybe TableInputFormatBase.getSplits is missing something similar to
>> HBASE-4109?
>>
>> J-D
>>
>> On Mon, Dec 17, 2012 at 5:26 PM, Robert Dyer <[EMAIL PROTECTED]> wrote:
>> > That's what I thought too.  Except I am running 0.94.2 and this fix was
>> > released in 0.90.4.
>> >
>> >
>> > On Mon, Dec 17, 2012 at 5:11 PM, Stack <[EMAIL PROTECTED]> wrote:
>> >
>> >> On Sun, Dec 16, 2012 at 1:16 AM, Robert Dyer <[EMAIL PROTECTED]> wrote:
>> >>
>> >>> I recently enabled reverse DNS on my test cluster.  Now when I run a
>> >>> MR
>> >>> job, the HBase input split locations are all adding a period to the
>> >>> end.
>> >>>  For example:
>> >>>
>> >>> /default-rack/foo-1.
>> >>> /default-rack/foo-2.
>> >>>
>> >>> Yet the machine locations are still correct:
>> >>>
>> >>> /default-rack/foo-1
>> >>> /default-rack/foo-2
>> >>>
>> >>> Since those strings don't match, it isn't assigning the tasks locally.
>> >>> It
>> >>> actually thinks 100% of the map tasks are rack-local and 0% data-local
>> >>> (although in reality, some still wind up being data-local due to sheer
>> >>> luck).
>> >>>
>> >>> What is the issue here?  Note that I don't have this problem with the
>> >>> MR
>> >>> tasks using SequenceFile as input, only with HBase's TableMapper.
>> >>>
>> >>
>> >>
>> >> Looks like https://issues.apache.org/jira/browse/HBASE-4109 ?
>> >> St.Ack
>> >>
>> >>
>
>
>
>
> --
>
> Robert Dyer
> [EMAIL PROTECTED]