Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Can I write to an compressed file which is located in hdfs?


Copy link to this message
-
Re: Can I write to an compressed file which is located in hdfs?
bejoy.hadoop@... 2012-02-06, 11:53
Hi
     If you have log files enough to become at least one block size in an hour. You can go ahead as
- run a scheduled job every hour that compresses the log files for that hour and stores them on to hdfs (can use LZO or even Snappy to compress)
- if your hive does more frequent analysis on this data store it as PARTITIONED BY (Date,Hour) . While loading into hdfs also follow a directory - sub dir structure. Once data is in hdfs issue a Alter Table Add Partition statement on corresponding hive table.
-in Hive DDL use the appropriate Input format (Hive has some ApacheLog Input Format already)
Regards
Bejoy K S

From handheld, Please excuse typos.

-----Original Message-----
From: Xiaobin She <[EMAIL PROTECTED]>
Date: Mon, 6 Feb 2012 16:41:50
To: <[EMAIL PROTECTED]>; 佘晓彬<[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: Can I write to an compressed file which is located in hdfs?

sorry, this sentence is wrong,

I can't compress these logs every hour and them put them into hdfs.

it should be

I can  compress these logs every hour and them put them into hdfs.
2012/2/6 Xiaobin She <[EMAIL PROTECTED]>

>
> hi all,
>
> I'm testing hadoop and hive, and I want to use them in log analysis.
>
> Here I have a question, can I write/append log to  an compressed file
> which is located in hdfs?
>
> Our system generate lots of log files every day, I can't compress these
> logs every hour and them put them into hdfs.
>
> But what if I want to write logs into files that was already in the hdfs
> and was compressed?
>
> Is these files were not compressed, then this job seems easy, but how to
> write or append logs into an compressed log?
>
> Can I do that?
>
> Can anyone give me some advices or give me some examples?
>
> Thank you very much!
>
> xiaobin
>