Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Roll based on date


Copy link to this message
-
Roll based on date
Hi,

Is there any option in HDFS sink that I can start rolling a new file
whenever the date in the log change? For example, I got below logs :

Oct 16 23:58:56 test-host : just test
Oct 16 23:59:51 test-host : test again
Oct 17 00:00:56 test-host : just test
Oct 17 00:00:56 test-host : test again

Then I want it to make a file on S3 bucket with result like this :

FlumeData.2013-10-16.1381916293017 <-- all the logs with Oct 16 from this
year 2013 will goes to here and when it's reach Oct 17 year 2013, then it
will start to sink into a new file below :

FlumeData.2013-10-17.1381940047117

Thanks.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB