Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Roll based on date

Copy link to this message
Roll based on date

Is there any option in HDFS sink that I can start rolling a new file
whenever the date in the log change? For example, I got below logs :

Oct 16 23:58:56 test-host : just test
Oct 16 23:59:51 test-host : test again
Oct 17 00:00:56 test-host : just test
Oct 17 00:00:56 test-host : test again

Then I want it to make a file on S3 bucket with result like this :

FlumeData.2013-10-16.1381916293017 <-- all the logs with Oct 16 from this
year 2013 will goes to here and when it's reach Oct 17 year 2013, then it
will start to sink into a new file below :