Using PigStorage() my pig script output gets put into partial files on the hadoop
file system.

When I use the copyToLocal fuction from Hadoop it creates a local directory with
all the partial files.

Is there a way to copy the partial files from hadoop into a single local file?

Thanks

Hi Aaron
         Instead of copyFromLocal use getmerge. It would do your job. The
syntax for CLI is
hadoop fs -getmerge <source dir in hdfs/pig output dir> <lfs destn
dir>/xyz.txt
Hope it helps!...

Regards
Bejoy.K.S

On Tue, Dec 6, 2011 at 1:57 AM, Aaron Griffith
<[EMAIL PROTECTED]>wrote:

> Using PigStorage() my pig script output gets put into partial files on the
> hadoop
> file system.
>
> When I use the copyToLocal fuction from Hadoop it creates a local
> directory with
> all the partial files.
>
> Is there a way to copy the partial files from hadoop into a single local
> file?
>
> Thanks
>
>

hadoop dfs cat /my/path/* > single_file

Russell Jurney
twitter.com/rjurney
[EMAIL PROTECTED]
datasyndrome.com

On Dec 5, 2011, at 12:30 PM, Aaron Griffith <[EMAIL PROTECTED]> wrote:

> Using PigStorage() my pig script output gets put into partial files on the hadoop
> file system.
>
> When I use the copyToLocal fuction from Hadoop it creates a local directory with
> all the partial files.
>
> Is there a way to copy the partial files from hadoop into a single local file?
>
> Thanks
>