Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Aggregate Warning does not work as expected on Embedding Pig in Java 0.9.1


Copy link to this message
-
Re: Aggregate Warning does not work as expected on Embedding Pig in Java 0.9.1
Hi Prashant,
I think it does make sense to enable warning aggregation even used
through PigServer.
Can you please open a jira ? (and a patch if possible !)

Thanks,
Thejas
On 12/8/11 11:28 PM, Prashant Kommireddi wrote:
> I noticed the property "aggregate.warning" is not being set by default when
> running PigServer, embedding Pig in Java.
>
> I was initially creating a PigServer object this way:
>   PigServer pigServer = new PigServer(ExecType.MAPREDUCE);
>
> But this generated detailed logs in the log directory, I noticed this only
> when the DNs ran out of disk space. I had to code around this to make it
> work:
>
> Properties properties = PropertiesUtil.loadDefaultProperties();
> properties.setProperty("aggregate.warning", "true");
> PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
>
> The behavior between Pig scripting and Embedded Pig be similar? Looking at
> the main constructor of PigServer, it looks like "aggregateWarning" is set
> to false if its not loaded in to Properties object.
>
> public PigServer(PigContext context, boolean connect) throws ExecException {
>          this.pigContext = context;
>          currDAG = new Graph(false);
>
>          aggregateWarning > "true".equalsIgnoreCase(pigContext.getProperties().getProperty("aggregate.warning"));
>          isMultiQuery > "true".equalsIgnoreCase(pigContext.getProperties().getProperty("opt.multiquery","true"));
>
>          jobName = pigContext.getProperties().getProperty(
>                  PigContext.JOB_NAME,
>                  PigContext.JOB_NAME_PREFIX + ":DefaultJobName");
>
>          if (connect) {
>              pigContext.connect();
>          }
>
>          addJarsFromProperties();
>      }
>
> Looks like a bug?
>