Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> hdfs.idleTimeout ,what's it used for ?


Copy link to this message
-
hdfs.idleTimeout ,what's it used for ?
Say If I have

a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/

hdfs.rollInterval=60

Now, if there is a file
/flume/events/2013-01-17/flume_XXXXXXXXX.tmp
This file is not ready to be rolled over yet, i.e. 60 seconds are not
up and now it's past 12 midnight, i.e. new day
And events start to be written to
/flume/events/2013-01-18/flume_XXXXXXXX.tmp

will the file 2013-01-17 never be rolled over, unless I have something
like hdfs.idleTimeout=60  ?
If so how do flume sinks keep track of files they need to rollover
after idealTimeout ?

In short what's the exact use of idealTimeout parameter ?