Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - hdfs.idleTimeout ,what's it used for ?


Copy link to this message
-
Re: hdfs.idleTimeout ,what's it used for ?
Juhani Connolly 2013-01-18, 06:37
>>
>> @Mohit:
>>
>> When flume dies unexpectedly the .tmp file remains. When it restarts
>> there is some logic in HDFS sink to recover it(and continue writing
>> from there). I'm not actually sure of the specifics. You may want to
>> try and just kill -9 a running flume process on a test machine and
>> then start it up, look at the logs and see what happens with the output.
>
> Does it also work when there is a long delay before flume gets
> started? We are bucketing by the hr so if start occurs in the next
> hour but flume actually died in previous hr and had  .tmp then does it
> still cleanup on restart

I'm not sure. I think your best bet here is to simulate this on a test
server. Start flume, after a bit kill 9 the process, wait until the
bucket becomes invalid, and restart.

My gut feeling is that it will recover if you have events with the
timestamp belonging to that bucket still incoming (in your persistent
channelor read in after recovery). If that path doesn't get touched
again though, it will probably remain as a .tmp file? *This could be
blatantly wrong, so I suggest you test it*