Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Aggregate Warning does not work as expected on Embedding Pig in Java 0.9.1


Copy link to this message
-
Re: Aggregate Warning does not work as expected on Embedding Pig in Java 0.9.1
Hi Prashant,
I think it does make sense to enable warning aggregation even used
through PigServer.
Can you please open a jira ? (and a patch if possible !)

Thanks,
Thejas
On 12/8/11 11:28 PM, Prashant Kommireddi wrote:
> I noticed the property "aggregate.warning" is not being set by default when
> running PigServer, embedding Pig in Java.
>
> I was initially creating a PigServer object this way:
>   PigServer pigServer = new PigServer(ExecType.MAPREDUCE);
>
> But this generated detailed logs in the log directory, I noticed this only
> when the DNs ran out of disk space. I had to code around this to make it
> work:
>
> Properties properties = PropertiesUtil.loadDefaultProperties();
> properties.setProperty("aggregate.warning", "true");
> PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
>
> The behavior between Pig scripting and Embedded Pig be similar? Looking at
> the main constructor of PigServer, it looks like "aggregateWarning" is set
> to false if its not loaded in to Properties object.
>
> public PigServer(PigContext context, boolean connect) throws ExecException {
>          this.pigContext = context;
>          currDAG = new Graph(false);
>
>          aggregateWarning > "true".equalsIgnoreCase(pigContext.getProperties().getProperty("aggregate.warning"));
>          isMultiQuery > "true".equalsIgnoreCase(pigContext.getProperties().getProperty("opt.multiquery","true"));
>
>          jobName = pigContext.getProperties().getProperty(
>                  PigContext.JOB_NAME,
>                  PigContext.JOB_NAME_PREFIX + ":DefaultJobName");
>
>          if (connect) {
>              pigContext.connect();
>          }
>
>          addJarsFromProperties();
>      }
>
> Looks like a bug?
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB