Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Pig counters with PigServer strange behavior


+
Charles Menguy 2011-12-06, 21:49
Copy link to this message
-
Re: Pig counters with PigServer strange behavior
There is a good blog article on this-
http://squarecog.wordpress.com/2010/12/24/incrementing-hadoop-counters-in-apache-pig/

Thanks,
Aniket

On Tue, Dec 6, 2011 at 1:49 PM, Charles Menguy <
[EMAIL PROTECTED]> wrote:

> Hi All,
>
> I'm trying to play with counters with PigServer and have a couple issues.
>
> First, I've found very little documentation on how to do this, so I'm not
> sure if the method I'm trying is the good one, any feedback would be
> appreciated.
>
> From what I understand, we need a PigStats in order to be able to retrieve
> the counters from it.
> To get this PigStats from a PigServer instance, here is what I do :
>
> pigServer.setBatchOn(); // needed to enable batch mode, which seems to be
> the only way to get the ExecJob instances needed to get the stats
> pigServer.registerScript(pigScript, params); // register the script i want
> to run
> List<ExecJob> execJobs = pigServer.executeBatch(); // get the ExecJobs
> associated with the script i just ran
>
> Now I am supposed to be able to get the counters from this ExecJob class.
>
> for (ExecJob execJob : execJobs) {
> for (JobStats jobStats : execJob.getStatistics().getJobGraph()) { // not
> sure why we need to use the job graph to get the stats but that seems to be
> the only solution i found
>        Counters counters = jobStats.getHadoopCounters(); // this is always
> NULL !
>  for (Group group : counters) {
> for (Counter counter : group) {
>
> Now the strange thing is that every time I call the getHadoopCounters(),
> the resulting Counters object is null, and thus I cannot get any counter at
> all.
>
> This happens in local and mapreduce mode, and I checked that execJob and
> jobStats are indeed not null.
>
> Am I doing something wrong here to get the counters, or forgetting
> something? I'm using pig 0.8.1 from cdh3u1
>
> Thanks for your help !
>
> --
> Charles Menguy | Senior Software Engineer
> Proclivity Systems
> 22 West 19th Street | 9th Floor
> New York, NY 10011
> [EMAIL PROTECTED]
> www.proclivitysystems.com
>
> Proclivity® | We Value Your Customers™
>
> This message is the property of Proclivity Systems, Inc. and is intended
> only for the use of the addressee(s), and may contain material that is
> confidential and privileged for the sole use of the intended recipient. If
> you are not the intended recipient, reliance or forwarding without express
> permission is strictly prohibited; please contact the sender and delete all
> copies.
>

--
"...:::Aniket:::... Quetzalco@tl"
+
Charles Menguy 2011-12-07, 15:04
+
Charles Menguy 2011-12-12, 20:35