Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Streaming reporter counter vs status


Copy link to this message
-
Streaming reporter counter vs status
Keith Wiley 2011-03-23, 18:02
I understand the basic distinction between a counter and a status even though both kinds of messages are sent to the reporter.  I'm particularly fond of the status messages because they permit me to observe the online behavior of numerous mappers (or reducers) simultaneously in the job tracker.  What I'm unclear about is which of these concepts (if either) is responsible for notifying the task tracker that the task is still alive and should not be killed after (ten I believe) minutes of inactivity.  If I only send, say, status messages, will the task still be killed after ten minutes?  If it is, in fact, a counter which is responsible for keeping the task alive (this has been my understanding and assumption so far), does it matter *which* counter I increment for the purpose of keeping the task alive or does any reporter:counter:a,b,c message keep the task tracker from killing the task?

As per my other post this morning, I am having serious problems keeping my tasks from being killed after ten minutes even though I am spawning a separate thread which does nothing except sleep for a minute and reporter counter and status messages forever.  Since the same tasks eventually succeed on the second or third try, I know the code *basically* works.  Otherwise none of the tasks would ever succeed I don't think.  Thus my vexation on this issue.

Thanks.

________________________________________________________________________________
Keith Wiley     [EMAIL PROTECTED]     keithwiley.com    music.keithwiley.com

"Luminous beings are we, not this crude matter."
                                           --  Yoda
________________________________________________________________________________