Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Pig counters with PigServer strange behavior

Copy link to this message
Re: Pig counters with PigServer strange behavior
There is a good blog article on this-


On Tue, Dec 6, 2011 at 1:49 PM, Charles Menguy <

> Hi All,
> I'm trying to play with counters with PigServer and have a couple issues.
> First, I've found very little documentation on how to do this, so I'm not
> sure if the method I'm trying is the good one, any feedback would be
> appreciated.
> From what I understand, we need a PigStats in order to be able to retrieve
> the counters from it.
> To get this PigStats from a PigServer instance, here is what I do :
> pigServer.setBatchOn(); // needed to enable batch mode, which seems to be
> the only way to get the ExecJob instances needed to get the stats
> pigServer.registerScript(pigScript, params); // register the script i want
> to run
> List<ExecJob> execJobs = pigServer.executeBatch(); // get the ExecJobs
> associated with the script i just ran
> Now I am supposed to be able to get the counters from this ExecJob class.
> for (ExecJob execJob : execJobs) {
> for (JobStats jobStats : execJob.getStatistics().getJobGraph()) { // not
> sure why we need to use the job graph to get the stats but that seems to be
> the only solution i found
>        Counters counters = jobStats.getHadoopCounters(); // this is always
> NULL !
>  for (Group group : counters) {
> for (Counter counter : group) {
> Now the strange thing is that every time I call the getHadoopCounters(),
> the resulting Counters object is null, and thus I cannot get any counter at
> all.
> This happens in local and mapreduce mode, and I checked that execJob and
> jobStats are indeed not null.
> Am I doing something wrong here to get the counters, or forgetting
> something? I'm using pig 0.8.1 from cdh3u1
> Thanks for your help !
> --
> Charles Menguy | Senior Software Engineer
> Proclivity Systems
> 22 West 19th Street | 9th Floor
> New York, NY 10011
> www.proclivitysystems.com
> Proclivity® | We Value Your Customers™
> This message is the property of Proclivity Systems, Inc. and is intended
> only for the use of the addressee(s), and may contain material that is
> confidential and privileged for the sole use of the intended recipient. If
> you are not the intended recipient, reliance or forwarding without express
> permission is strictly prohibited; please contact the sender and delete all
> copies.

"...:::Aniket:::... Quetzalco@tl"