Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Pig counters with PigServer strange behavior


+
Charles Menguy 2011-12-06, 21:49
Copy link to this message
-
Re: Pig counters with PigServer strange behavior
There is a good blog article on this-
http://squarecog.wordpress.com/2010/12/24/incrementing-hadoop-counters-in-apache-pig/

Thanks,
Aniket

On Tue, Dec 6, 2011 at 1:49 PM, Charles Menguy <
[EMAIL PROTECTED]> wrote:

> Hi All,
>
> I'm trying to play with counters with PigServer and have a couple issues.
>
> First, I've found very little documentation on how to do this, so I'm not
> sure if the method I'm trying is the good one, any feedback would be
> appreciated.
>
> From what I understand, we need a PigStats in order to be able to retrieve
> the counters from it.
> To get this PigStats from a PigServer instance, here is what I do :
>
> pigServer.setBatchOn(); // needed to enable batch mode, which seems to be
> the only way to get the ExecJob instances needed to get the stats
> pigServer.registerScript(pigScript, params); // register the script i want
> to run
> List<ExecJob> execJobs = pigServer.executeBatch(); // get the ExecJobs
> associated with the script i just ran
>
> Now I am supposed to be able to get the counters from this ExecJob class.
>
> for (ExecJob execJob : execJobs) {
> for (JobStats jobStats : execJob.getStatistics().getJobGraph()) { // not
> sure why we need to use the job graph to get the stats but that seems to be
> the only solution i found
>        Counters counters = jobStats.getHadoopCounters(); // this is always
> NULL !
>  for (Group group : counters) {
> for (Counter counter : group) {
>
> Now the strange thing is that every time I call the getHadoopCounters(),
> the resulting Counters object is null, and thus I cannot get any counter at
> all.
>
> This happens in local and mapreduce mode, and I checked that execJob and
> jobStats are indeed not null.
>
> Am I doing something wrong here to get the counters, or forgetting
> something? I'm using pig 0.8.1 from cdh3u1
>
> Thanks for your help !
>
> --
> Charles Menguy | Senior Software Engineer
> Proclivity Systems
> 22 West 19th Street | 9th Floor
> New York, NY 10011
> [EMAIL PROTECTED]
> www.proclivitysystems.com
>
> Proclivity® | We Value Your Customers™
>
> This message is the property of Proclivity Systems, Inc. and is intended
> only for the use of the addressee(s), and may contain material that is
> confidential and privileged for the sole use of the intended recipient. If
> you are not the intended recipient, reliance or forwarding without express
> permission is strictly prohibited; please contact the sender and delete all
> copies.
>

--
"...:::Aniket:::... Quetzalco@tl"
+
Charles Menguy 2011-12-07, 15:04
+
Charles Menguy 2011-12-12, 20:35
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB