Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> for UDF, figure out whether it's on a task tracker?


Copy link to this message
-
Re: for UDF, figure out whether it's on a task tracker?
I see, thanks Jonathan.

On Tue, Jul 3, 2012 at 10:01 AM, Jonathan Coveney <[EMAIL PROTECTED]>wrote:

> UDF's are instantiated at job construction time a couple of times in order
> to inspect various properties about them. This is subideal, but alas. I
> generally lazily initialize in exec, as that is only called on the
> mapper/reducer. The lifecycle of UDF's can be a bit confusing in this way.
>
> 2012/7/3 Yang <[EMAIL PROTECTED]>
>
> > normally job tracker and task tracker is on different nodes.
> >
> > when I submit a pig script using UDF. I think the UDF constructor is
> first
> > run (several times, don't know why)
> > on the job tracker, and then it's run on each of the task trackers.
> >
> > now I want to do some custom work inside the constructor, such as
> checking
> > the existence of certain files
> > which are specific to only task trackers. such work only needs to be done
> > on task trackers.
> > So , is there a way to figure out whether the UDF is being run on task
> > tracker or job tracker?
> >
> > Thanks!
> > yang
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB