Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> streaming question.


Copy link to this message
-
Re: streaming question.
Dmitry,

If you are talking about Text data, then the splits can be anywhere.  But
LineRecordReader will take care of this thing and your mapper code will
get the correct whole line.

Abdul Qadeer

On Sun, Jan 18, 2009 at 9:59 AM, Dmitry Pushkarev <[EMAIL PROTECTED]> wrote:

> Dear hadoop users.
>
>
>
> When I use streaming on one large file, that is being split in many map
> tasks, can I be sure that splits won't fall in the middle of the line?
>
> (i.e. if split size needs to be larger than  64Mb to fit end of the line it
> will be increased?
>
>
>
> Thanks.
>
> ---
>
> Dmitry Pushkarev
>
> +1-650-644-8988
>
>
>
>