Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Best way to get data out of HDFS to other platforms


Copy link to this message
-
Re: Best way to get data out of HDFS to other platforms
Mapred Learn 2012-04-13, 20:24
This is for copying to Linux locally
I m talking about cross platform where you want to get data to systems like AIX where these APIs don't work.

Sent from my iPhone

On Apr 13, 2012, at 1:02 PM, Dave Shine <[EMAIL PROTECTED]> wrote:

> fs.copyToLocal(sourcePath, destinationPath);
>
>  
>
> Am I missing something???
>
>  
>
> Dave
>
>  
>
>  
>
> From: Mac Noland [mailto:[EMAIL PROTECTED]]
> Sent: Friday, April 13, 2012 2:59 PM
> To: [EMAIL PROTECTED]; CDH Users
> Subject: Re: Best way to get data out of HDFS to other platforms
>
>  
>
> There are probably much, much, much better and more standard ways, but in the simple push and pulls I've done, I've just rolled my own.
>
>  
>
>     public void copyFileFromHDFS(String src, String dest, boolean overWrite) throws IOException{
>
>         Path hdfsPath = new Path(src);
>         FSDataInputStream hdfsFileInPut = fs.open(hdfsPath);
>        
>         byte[] data = new byte[1024];
>         File localFile = new File(dest);
>        
>         if (overWrite && localFile.exists()){
>             localFile.delete();
>         }
>        
>         FileOutputStream localFileOutPut = new FileOutputStream(localFile);
>         int off;
>        
>         while ((off = hdfsFileInPut.read(data)) > -1){
>             localFileOutPut.write(data,0,off);
>         }
>        
>         hdfsFileInPut.close();
>         localFileOutPut.close();
>
>     }
>
>  
>
> From: Mapred Learn <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; CDH Users <[EMAIL PROTECTED]>
> Sent: Friday, April 13, 2012 10:29 AM
> Subject: Best way to get data out of HDFS to other platforms
>
>
>
> Hi All,
>
> I was trying to find out the best options to get data out of HDFS to other platforms like AIX with parameters like reliability, performance and parallelism and data integrity in mind.
>
>
> Thanks,
> Sent from my iPhone
>
>
> The information contained in this email message is considered confidential and proprietary to the sender and is intended solely for review and use by the named recipient. Any unauthorized review, use or distribution is strictly prohibited. If you have received this message in error, please advise the sender by reply email and delete the message.