Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Task status query


Copy link to this message
-
Re: Task status query
Right now its MR specific (TaskUmbilicalProtocol) - YARN doesn't have
any reusable items here yet, but there are easy to use RPC libs such
as Avro and Thrift out there that make it easy to do such things once
you define what you want in a schema/spec form.

On Fri, Sep 20, 2013 at 5:32 PM, John Lilley <[EMAIL PROTECTED]> wrote:
> Thanks Harsh.  Is this protocol something that is available to all AMs/tasks?  Or is it up to each AM/task pair to develop their own protocol?
> john
>
> -----Original Message-----
> From: Harsh J [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, September 19, 2013 9:20 PM
> To: <[EMAIL PROTECTED]>
> Subject: Re: Task status query
>
> Hi John,
>
> YARN tasks can be more than simple executables. In case of MR, for example, tasks talk to the AM and report their individual progress and counters back to it, via a specific protocol (over the network), giving the AM more data to compute an near-accurate global progress.
>
> On Fri, Sep 20, 2013 at 12:18 AM, John Lilley <[EMAIL PROTECTED]> wrote:
>> How does a YARN application master typically query ongoing status
>> (like percentage completion) of its tasks?
>>
>> I would like to be able to ultimately relay information to the user like:
>>
>> 100 tasks are scheduled
>>
>> 10 tasks are complete
>>
>> 4 tasks are running and they are (4%, 10%, 50%, 70%) complete
>>
>> But, given that YARN tasks are simply executables, how can the AM even
>> get at this information?  Can the AM get access to stdout/stderr?
>>
>> Thanks
>>
>> John
>>
>>
>
>
>
> --
> Harsh J

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB