Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Quick question


Copy link to this message
-
Re: Quick question
Ted Dunning 2011-02-18, 19:25
The input is effectively split by lines, but under the covers, the actual
splits are by byte.  Each mapper will cleverly scan from the specified start
to the next line after the start point.  At then end, it will over-read to
the end of line that is at or after the end of its specified region.  This
can make the last split be a bit smaller than the others and the first be a
bit larger.

Practically speaking, however, your 2000 line file is extremely unlikely to
be split at all because it is sooo small.

On Fri, Feb 18, 2011 at 11:14 AM, maha <[EMAIL PROTECTED]> wrote:

> Hi all,
>
>  I want to check if the following statement is right:
>
>  If I use TextInputFormat to process a text file with 2000 lines (each
> ending with \n) with 20 mappers. Then each map will have a sequence of
> COMPLETE LINES .
>
> In other words,  the input is not split byte-wise but by lines.
>
> Is that right?
>
>
> Thank you,
> Maha