Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - streaming question.


+
Dmitry Pushkarev 2009-01-18, 04:59
Copy link to this message
-
Re: streaming question.
Abdul Qadeer 2009-01-18, 09:43
Dmitry,

If you are talking about Text data, then the splits can be anywhere.  But
LineRecordReader will take care of this thing and your mapper code will
get the correct whole line.

Abdul Qadeer

On Sun, Jan 18, 2009 at 9:59 AM, Dmitry Pushkarev <[EMAIL PROTECTED]> wrote:

> Dear hadoop users.
>
>
>
> When I use streaming on one large file, that is being split in many map
> tasks, can I be sure that splits won't fall in the middle of the line?
>
> (i.e. if split size needs to be larger than  64Mb to fit end of the line it
> will be increased?
>
>
>
> Thanks.
>
> ---
>
> Dmitry Pushkarev
>
> +1-650-644-8988
>
>
>
>
+
Amareshwari Sriramadasu 2009-01-19, 03:51