Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Reg: parsing all files & file append


Copy link to this message
-
Re: Reg: parsing all files & file append
Manoj Babu 2012-09-10, 05:33
Thank you Bejoy.

Does file append is production stable?
Cheers!
Manoj.

On Sun, Sep 9, 2012 at 10:19 PM, Bejoy KS <[EMAIL PROTECTED]> wrote:

> **
> Hi Manoj
>
> You can load daily logs into a individual directories in hdfs and process
> them daily. Keep those results in hdfs or hbase or dbs etc. Every day do
> the processing, get the results and aggregate the same with the previously
> aggregated results till date.
>
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
> ------------------------------
> *From: * Manoj Babu <[EMAIL PROTECTED]>
> *Date: *Sun, 9 Sep 2012 21:28:54 +0530
> *To: *<[EMAIL PROTECTED]>
> *ReplyTo: * [EMAIL PROTECTED]
> *Subject: *Reg: parsing all files & file append
>
> Hi All,
>
> I have two questions, providing info on it will be helpful.
>
> 1, I am using hadoop to analyze and to find top n search term metric's
> from logs.
> If any new log file is added to HDFS then again we are running the job to
> find the metrics.
> Daily we will be getting log files and we are parsing the whole file and
> getting the metric's.
> All the log file's are parsed daily to get the latest metric's is there
> any way is there any way to avoid this?
>
> 2, Does file append is production stable?
>
> Cheers!
> Manoj.
>
>