Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: copy chunk of hadoop output


Copy link to this message
-
Re: copy chunk of hadoop output
But be careful.

hadoop fs -cat will retrieve the entire file and last only when it
will have retrieve the last bytes you are looking for.

If your file is many GB big, it will take a lot of time for this
command to complete and will put some pressure on your network.

JM

2013/2/19, jamal sasha <[EMAIL PROTECTED]>:
> Awesome thanks :)
>
>
> On Tue, Feb 19, 2013 at 2:14 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> You can instead use 'fs -cat' and the 'head' coreutil, as one example:
>>
>> hadoop fs -cat 100-byte-dfs-file | head -c 5 > 5-byte-local-file
>>
>> On Wed, Feb 20, 2013 at 3:38 AM, jamal sasha <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> >   I was wondering in the following command:
>> >
>> > bin/hadoop dfs -copyToLocal hdfspath localpath
>> > can we have specify to copy not full but like xMB's of file to local
>> drive?
>> >
>> > Is something like this possible
>> > Thanks
>> > Jamal
>>
>>
>>
>> --
>> Harsh J
>>
>