Hi,

I need to upload large xml files files daily. Right now am having a small
program to read all the files from local folder and writing it to HDFS as a
single file. Is this a right way?
If there any best practices or optimized way to achieve this Kindly let me
know.

Thanks in advance!

Cheers!
Manoj.

Hi Manoj

If you are looking at a scheduler and a work flow manager to carry out this task you can have a look at oozie.

If your xml files are smaller(smaller than hdfs block size) then definitely it is a better practice to combine them to form larger files. Combining into Sequence Files should be good.

Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: Manoj Babu <[EMAIL PROTECTED]>
Date: Fri, 13 Jul 2012 08:59:51
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: suggest Best way to upload xml files to HDFS

Hi,

I need to upload large xml files files daily. Right now am having a small
program to read all the files from local folder and writing it to HDFS as a
single file. Is this a right way?
If there any best practices or optimized way to achieve this Kindly let me
know.

Thanks in advance!

Cheers!
Manoj.

Hi,

Could you kindly provide the pros and cons of Multifile, combilefile,
sequencefile input format?

Thanks in Advance.

Cheers!
Manoj.

On Fri, Jul 13, 2012 at 10:15 AM, Bejoy KS <[EMAIL PROTECTED]> wrote:

> **
> Hi Manoj
>
> If you are looking at a scheduler and a work flow manager to carry out
> this task you can have a look at oozie.
>
> If your xml files are smaller(smaller than hdfs block size) then
> definitely it is a better practice to combine them to form larger files.
> Combining into Sequence Files should be good.
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
> ------------------------------
> *From: * Manoj Babu <[EMAIL PROTECTED]>
> *Date: *Fri, 13 Jul 2012 08:59:51 +0530
> *To: *<[EMAIL PROTECTED]>
> *ReplyTo: * [EMAIL PROTECTED]
> *Subject: *suggest Best way to upload xml files to HDFS
>
> Hi,
>
> I need to upload large xml files files daily. Right now am having a small
> program to read all the files from local folder and writing it to HDFS as a
> single file. Is this a right way?
> If there any best practices or optimized way to achieve this Kindly let me
> know.
>
> Thanks in advance!
>
> Cheers!
> Manoj.
>
>

If you're looking at automated file/record/event collection, take a
look at Apache Flume: http://incubator.apache.org/flume/. It does well
for distributed collections as well and is very configurable.

Otherwise, write a scheduled script to do the uploads every X period
(your choice). Consider using
https://github.com/edwardcapriolo/filecrush or similar tools too, if
your files are much small and getting in the way of MR processing.

On Fri, Jul 13, 2012 at 8:59 AM, Manoj Babu <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I need to upload large xml files files daily. Right now am having a small
> program to read all the files from local folder and writing it to HDFS as a
> single file. Is this a right way?
> If there any best practices or optimized way to achieve this Kindly let me
> know.
>
> Thanks in advance!
>
> Cheers!
> Manoj.
>

--
Harsh J