Hi,

I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

Thanks,
-Rakesh
     

Not at this point as parameter substitution is implemented a s preprocessor on the script.

Olga

-----Original Message-----
From: rakesh kothari [mailto:[EMAIL PROTECTED]]
Sent: Thursday, October 07, 2010 11:47 AM
To: [EMAIL PROTECTED]
Subject: Passing parameters to Pig Script using Java
Hi,

I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

Thanks,
-Rakesh
     

Hi,

Have a look at the grunt source code, you'll need to pass your script through the preprocess stage with thr parameters to create a substituted pig file, which you can then pass to pig. I've written code that does this before by just doing what grunt does.

Cheers,

----- Original Message -----
From: Olga Natkovich <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED] <[EMAIL PROTECTED]>; [EMAIL PROTECTED] <[EMAIL PROTECTED]>
Sent: Thu Oct 07 21:27:58 2010
Subject: RE: Passing parameters to Pig Script using Java

Not at this point as parameter substitution is implemented a s preprocessor on the script.

Olga

-----Original Message-----
From: rakesh kothari [mailto:[EMAIL PROTECTED]]
Sent: Thursday, October 07, 2010 11:47 AM
To: [EMAIL PROTECTED]
Subject: Passing parameters to Pig Script using Java
Hi,

I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

Thanks,
-Rakesh
     

Here's my workaround:
I extend PigServer with the following code copy/pasted from other places in Pig.
Parameters are in the form: "foo=bar"
Julien

import org.apache.pig.ExecType;
import org.apache.pig.backend.executionengine.ExecException;
import org.apache.pig.backend.executionengine.ExecJob;
import org.apache.pig.impl.PigContext;
import org.apache.pig.tools.grunt.GruntParser;
import org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor;
    /**
     * adapted from org.apache.pig.Main
     * returns the stream of final pig script to be passed to Grunt
     */
    private Reader runParamPreprocessor(InputStream origPigScript, List<String> params,
                                               List<String> paramFiles, String scriptFile)
                                    throws org.apache.pig.tools.parameters.ParseException, IOException{
        ParameterSubstitutionPreprocessor psp = new ParameterSubstitutionPreprocessor(50);
        String[] type1 = new String[1];
        String[] type2 = new String[1];

        StringWriter writer = new StringWriter();
        psp.genSubstitutedFile (new BufferedReader(new InputStreamReader(origPigScript)), writer,  params.size() > 0 ? params.toArray(type1) : null,
          paramFiles.size() > 0 ? paramFiles.toArray(type2) : null);

        return new BufferedReader(new StringReader(writer.toString()));

    }

    /**
  * adapted original code from file to apply the preprocessor
  * @param filePath relative path of the pig script
  * @param params parameters to be applied by the preprocessor
  * @throws IOException
  */
    public void registerScript(String filePath, List<String> params) throws IOException {
        try {
            InputStream stream = new FileInputStream(filePath);
            try {
             GruntParser grunt = new GruntParser(runParamPreprocessor(stream, params, new ArrayList<String>(), filePath));
             grunt.setInteractive(false);
             grunt.setParams(this);
             grunt.parseStopOnError(true);
            } finally {
             try {
              stream.close();
             } catch (IOException e) {
              e.printStackTrace();
             }
            }
        } catch (org.apache.pig.tools.pigscript.parser.ParseException e) {
            throw new IOException("Error while parsing script: "+filePath,e);
        } catch (org.apache.pig.tools.parameters.ParseException e) {
         throw new IOException("Error while parsing parameters: "+params,e);
  }
    }

    /**
  * adapted original code from file to apply the preprocessor
  * @param filePath relative path of the pig script
  * @param params parameters to be applied by the preprocessor
  * @throws IOException
  */
    public void registerScript(String filePath, String... params) throws IOException {
     registerScript(filePath, Arrays.asList(params));
    }

On 10/7/10 1:27 PM, "Olga Natkovich" <[EMAIL PROTECTED]> wrote:

Not at this point as parameter substitution is implemented a s preprocessor on the script.

Olga

-----Original Message-----
From: rakesh kothari [mailto:[EMAIL PROTECTED]]
Sent: Thursday, October 07, 2010 11:47 AM
To: [EMAIL PROTECTED]
Subject: Passing parameters to Pig Script using Java
Hi,

I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

Thanks,
-Rakesh

Hi Julien,

You did what we have done in Pig 0.8. There's a little difference
between your api and ours. We encapsulates the parameter in Map rather
than List.
On Fri, Oct 8, 2010 at 4:43 AM, Julien Le Dem <[EMAIL PROTECTED]> wrote:
> Here's my workaround:
> I extend PigServer with the following code copy/pasted from other places in Pig.
> Parameters are in the form: "foo=bar"
> Julien
>
> import org.apache.pig.ExecType;
> import org.apache.pig.backend.executionengine.ExecException;
> import org.apache.pig.backend.executionengine.ExecJob;
> import org.apache.pig.impl.PigContext;
> import org.apache.pig.tools.grunt.GruntParser;
> import org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor;
>
>
>    /**
>     * adapted from org.apache.pig.Main
>     * returns the stream of final pig script to be passed to Grunt
>     */
>    private Reader runParamPreprocessor(InputStream origPigScript, List<String> params,
>                     ��                         List<String> paramFiles, String scriptFile)
>                     ��              throws org.apache.pig.tools.parameters.ParseException, IOException{
>        ParameterSubstitutionPreprocessor psp = new ParameterSubstitutionPreprocessor(50);
>        String[] type1 = new String[1];
>        String[] type2 = new String[1];
>
>        StringWriter writer = new StringWriter();
>        psp.genSubstitutedFile (new BufferedReader(new InputStreamReader(origPigScript)), writer,  params.size() > 0 ? params.toArray(type1) : null,
>          paramFiles.size() > 0 ? paramFiles.toArray(type2) : null);
>
>        return new BufferedReader(new StringReader(writer.toString()));
>
>    }
>
>    /**
>  * adapted original code from file to apply the preprocessor
>  * @param filePath relative path of the pig script
>  * @param params parameters to be applied by the preprocessor
>  * @throws IOException
>  */
>    public void registerScript(String filePath, List<String> params) throws IOException {
>        try {
>            InputStream stream = new FileInputStream(filePath);
>            try {
>             GruntParser grunt = new GruntParser(runParamPreprocessor(stream, params, new ArrayList<String>(), filePath));
>             grunt.setInteractive(false);
>             grunt.setParams(this);
>             grunt.parseStopOnError(true);
>            } finally {
>             try {
>              stream.close();
>             } catch (IOException e) {
>              e.printStackTrace();
>             }
>            }
>        } catch (org.apache.pig.tools.pigscript.parser.ParseException e) {
>            throw new IOException("Error while parsing script: "+filePath,e);
>        } catch (org.apache.pig.tools.parameters.ParseException e) {
>         throw new IOException("Error while parsing parameters: "+params,e);
>  }
>    }
>
>    /**
>  * adapted original code from file to apply the preprocessor
>  * @param filePath relative path of the pig script
>  * @param params parameters to be applied by the preprocessor
>  * @throws IOException
>  */
>    public void registerScript(String filePath, String... params) throws IOException {
>     registerScript(filePath, Arrays.asList(params));
>    }
>
>
>
>
>
> On 10/7/10 1:27 PM, "Olga Natkovich" <[EMAIL PROTECTED]> wrote:
>
> Not at this point as parameter substitution is implemented a s preprocessor on the script.
>
> Olga
>
> -----Original Message-----
> From: rakesh kothari [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, October 07, 2010 11:47 AM
> To: [EMAIL PROTECTED]
> Subject: Passing parameters to Pig Script using Java
>
>
> Hi,
>
> I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?
>
> Thanks,
> -Rakesh
>
>
>

--
Best Regards

Jeff Zhang

Oops, sorry
Yes this is old code.
Thanks for pointing this out.
This should give a hint though.
Julien
On 10/7/10 6:16 PM, "Jeff Zhang" <[EMAIL PROTECTED]> wrote:

Hi Julien,

You did what we have done in Pig 0.8. There's a little difference
between your api and ours. We encapsulates the parameter in Map rather
than List.
On Fri, Oct 8, 2010 at 4:43 AM, Julien Le Dem <[EMAIL PROTECTED]> wrote:
> Here's my workaround:
> I extend PigServer with the following code copy/pasted from other places in Pig.
> Parameters are in the form: "foo=bar"
> Julien
>
> import org.apache.pig.ExecType;
> import org.apache.pig.backend.executionengine.ExecException;
> import org.apache.pig.backend.executionengine.ExecJob;
> import org.apache.pig.impl.PigContext;
> import org.apache.pig.tools.grunt.GruntParser;
> import org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor;
>
>
>    /**
>     * adapted from org.apache.pig.Main
>     * returns the stream of final pig script to be passed to Grunt
>     */
>    private Reader runParamPreprocessor(InputStream origPigScript, List<String> params,
>                                               List<String> paramFiles, String scriptFile)
>                                    throws org.apache.pig.tools.parameters.ParseException, IOException{
>        ParameterSubstitutionPreprocessor psp = new ParameterSubstitutionPreprocessor(50);
>        String[] type1 = new String[1];
>        String[] type2 = new String[1];
>
>        StringWriter writer = new StringWriter();
>        psp.genSubstitutedFile (new BufferedReader(new InputStreamReader(origPigScript)), writer,  params.size() > 0 ? params.toArray(type1) : null,
>          paramFiles.size() > 0 ? paramFiles.toArray(type2) : null);
>
>        return new BufferedReader(new StringReader(writer.toString()));
>
>    }
>
>    /**
>  * adapted original code from file to apply the preprocessor
>  * @param filePath relative path of the pig script
>  * @param params parameters to be applied by the preprocessor
>  * @throws IOException
>  */
>    public void registerScript(String filePath, List<String> params) throws IOException {
>        try {
>            InputStream stream = new FileInputStream(filePath);
>            try {
>             GruntParser grunt = new GruntParser(runParamPreprocessor(stream, params, new ArrayList<String>(), filePath));
>             grunt.setInteractive(false);
>             grunt.setParams(this);
>             grunt.parseStopOnError(true);
>            } finally {
>             try {
>              stream.close();
>             } catch (IOException e) {
>              e.printStackTrace();
>             }
>            }
>        } catch (org.apache.pig.tools.pigscript.parser.ParseException e) {
>            throw new IOException("Error while parsing script: "+filePath,e);
>        } catch (org.apache.pig.tools.parameters.ParseException e) {
>         throw new IOException("Error while parsing parameters: "+params,e);
>  }
>    }
>
>    /**
>  * adapted original code from file to apply the preprocessor
>  * @param filePath relative path of the pig script
>  * @param params parameters to be applied by the preprocessor
>  * @throws IOException
>  */
>    public void registerScript(String filePath, String... params) throws IOException {
>     registerScript(filePath, Arrays.asList(params));
>    }
>
>
>
>
>
> On 10/7/10 1:27 PM, "Olga Natkovich" <[EMAIL PROTECTED]> wrote:
>
> Not at this point as parameter substitution is implemented a s preprocessor on the script.
>
> Olga
>
> -----Original Message-----
> From: rakesh kothari [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, October 07, 2010 11:47 AM
> To: [EMAIL PROTECTED]
> Subject: Passing parameters to Pig Script using Java
>
>
> Hi,
>
> I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

Best Regards

Jeff Zhang