Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: HDFS File Appending URGENT


Copy link to this message
-
Fw: HDFS File Appending URGENT
Please help me on this.
I need it very urgently

Regards,
Jagaran
----- Forwarded Message ----
From: jagaran das <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Thu, 16 June, 2011 9:51:51 PM
Subject: Re: HDFS File Appending URGENT

Thanks a lot Xiabo.

I have tried with the  below code in HDFS version 0.20.20 and it worked.
Is it not stable yet?

public class HadoopFileWriter {
public static void main (String [] args) throws Exception{
try{
URI uri = new
URI("hdfs://localhost:9000/Users/jagarandas/Work-Assignment/Analytics/analytics-poc/hadoop-0.20.203.0/data/test.dat");
Path pt=new Path(uri);
FileSystem fs = FileSystem.get(new Configuration());
BufferedWriter br;
if(fs.isFile(pt)){
br=new BufferedWriter(new OutputStreamWriter(fs.append(pt)));
br.newLine();
}else{
br=new BufferedWriter(new OutputStreamWriter(fs.create(pt,true)));
}
String line = args[0];
System.out.println(line);
br.write(line);
br.close();
}catch(Exception e){
e.printStackTrace();
System.out.println("File not found");
}
}
}

Thanks a lot for your help.

Regards,
Jagaran
________________________________
From: Xiaobo Gu <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Thu, 16 June, 2011 8:01:14 PM
Subject: Re: HDFS File Appending URGENT

You can merge multiple files into a new one, there is no means to
append to a existing file.

On Fri, Jun 17, 2011 at 10:29 AM, jagaran das <[EMAIL PROTECTED]> wrote:
> Is the hadoop version Hadoop 0.20.203.0 API
>
> That means still the hadoop files in HDFS version 0.20.20  are immutable?
> And there is no means we can append to an existing file in HDFS?
>
> We need to do this urgently as we have do set up the pipeline accordingly in
> production?
>
> Regards,
> Jagaran
>
>
>
> ________________________________
> From: Xiaobo Gu <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Thu, 16 June, 2011 6:26:45 PM
> Subject: Re: HDFS File Appending
>
> please refer to FileUtil.CopyMerge
>
> On Fri, Jun 17, 2011 at 8:33 AM, jagaran das <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> We have a requirement where
>>
>>  There would be huge number of small files to be pushed to hdfs and then use
>>pig
>> to do analysis.
>>  To get around the classic "Small File Issue" we merge the files and push a
>> bigger file in to HDFS.
>>  But we are loosing time in this merging process of our pipeline.
>>
>> But If we can directly append to an existing file in HDFS we can save this
>> "Merging Files" time.
>>
>> Can you please suggest if there a newer stable version of Hadoop where can go
>> for appending ?
>>
>> Thanks and Regards,
>> Jagaran
>