Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> little help on reading debug output


Copy link to this message
-
little help on reading debug output
Is there any documentation on how to read this output when I 'set debug on' I get in my reducer syslog:

DEBUG: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce - New For Each(true,true)[tuple] - 1-770
|   |
|   POBinCond[bag] - 1-768
|   |
|   |---Project[bag][1] - 1-764
|   |
|   |---POUserFunc(org.apache.pig.builtin.IsEmpty)[boolean] - 1-766
|   |   |
|   |   |---Project[bag][1] - 1-765
|   |
|   |---Constant({()}) - 1-767
|   |
|   Project[bag][2] - 1-769
DEBUG: org.apache.pig.data.InternalCachedBag - Memory can hold 45450 records, put the rest in spill file.
DEBUG: org.apache.pig.data.InternalCachedBag - Memory can hold 45192 records, put the rest in spill file.
DEBUG: org.apache.pig.data.InternalCachedBag - Memory can hold 44852 records, put the rest in spill file

Specifically what do the 1-7** numbers mean?  Is it possible to get line numbers from the pig script :)
Also strange is that it seems that POUserFunc is telling me we are running the IsEmpty UDF but that UDF isn't being called in this script at all...is it possible pig is using it under the covers?
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB