Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> High level technical overview of output bucketing for flume (old-gen) ?


Copy link to this message
-
Re: High level technical overview of output bucketing for flume (old-gen) ?
Hi Bertrand,

I have written a blog about in 2011, here you can see for what you can see the use of bucketing:
http://mapredit.blogspot.de/2011/10/centralized-logfile-management-across.html

You can use the sequences to create directories, based on the sequences the timestamp on a syslog event will be delivered. So you have the availability to automatically create directories for year, month, day, hour or something like that.

Best,
 Alex

On Jan 4, 2013, at 3:22 PM, Bertrand Dechoux <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am using flume (old gen) as an extension to an existant syslog system and
> would like to use the timestamp of the syslog message as the timestamp of
> the flume event.
> I guess the timestamp is used for the '*Fine grained escape sequences date
> and times*' but I don't have a clear understanding of it.
> http://archive.cloudera.com/cdh/3/flume/UserGuide/#_output_bucketing
>
> Could someone point me to where those sequences (like %d) are interpreted?
> I would like to be sure I am not missing anything obvious.
>
> Thanks in advance
>
> Bertrand
>
> PS : I know an unrelated recommandation would be to use flume-ng but this
> is not the topic of this email.

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF