Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> hdfs.idleTimeout ,what's it used for ?


+
Bhaskar V. Karambelkar 2013-01-17, 20:07
+
Connor Woodson 2013-01-17, 20:29
+
Bhaskar V. Karambelkar 2013-01-17, 21:38
+
Juhani Connolly 2013-01-18, 02:08
+
Mohit Anchlia 2013-01-18, 02:17
+
Connor Woodson 2013-01-18, 02:19
+
Connor Woodson 2013-01-18, 02:20
+
Connor Woodson 2013-01-18, 02:23
+
Juhani Connolly 2013-01-18, 02:46
+
Connor Woodson 2013-01-18, 03:24
+
Juhani Connolly 2013-01-18, 03:39
+
Connor Woodson 2013-01-18, 04:18
+
Mohit Anchlia 2013-01-18, 05:12
Copy link to this message
-
Re: hdfs.idleTimeout ,what's it used for ?
>>
>> @Mohit:
>>
>> When flume dies unexpectedly the .tmp file remains. When it restarts
>> there is some logic in HDFS sink to recover it(and continue writing
>> from there). I'm not actually sure of the specifics. You may want to
>> try and just kill -9 a running flume process on a test machine and
>> then start it up, look at the logs and see what happens with the output.
>
> Does it also work when there is a long delay before flume gets
> started? We are bucketing by the hr so if start occurs in the next
> hour but flume actually died in previous hr and had  .tmp then does it
> still cleanup on restart

I'm not sure. I think your best bet here is to simulate this on a test
server. Start flume, after a bit kill 9 the process, wait until the
bucket becomes invalid, and restart.

My gut feeling is that it will recover if you have events with the
timestamp belonging to that bucket still incoming (in your persistent
channelor read in after recovery). If that path doesn't get touched
again though, it will probably remain as a .tmp file? *This could be
blatantly wrong, so I suggest you test it*
+
Juhani Connolly 2013-01-18, 02:39
+
Connor Woodson 2013-01-18, 02:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB