Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Passing parameters to Pig Script using Java


Copy link to this message
-
Re: Passing parameters to Pig Script using Java
Julien Le Dem 2010-10-07, 20:43
Here's my workaround:
I extend PigServer with the following code copy/pasted from other places in Pig.
Parameters are in the form: "foo=bar"
Julien

import org.apache.pig.ExecType;
import org.apache.pig.backend.executionengine.ExecException;
import org.apache.pig.backend.executionengine.ExecJob;
import org.apache.pig.impl.PigContext;
import org.apache.pig.tools.grunt.GruntParser;
import org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor;
    /**
     * adapted from org.apache.pig.Main
     * returns the stream of final pig script to be passed to Grunt
     */
    private Reader runParamPreprocessor(InputStream origPigScript, List<String> params,
                                               List<String> paramFiles, String scriptFile)
                                    throws org.apache.pig.tools.parameters.ParseException, IOException{
        ParameterSubstitutionPreprocessor psp = new ParameterSubstitutionPreprocessor(50);
        String[] type1 = new String[1];
        String[] type2 = new String[1];

        StringWriter writer = new StringWriter();
        psp.genSubstitutedFile (new BufferedReader(new InputStreamReader(origPigScript)), writer,  params.size() > 0 ? params.toArray(type1) : null,
          paramFiles.size() > 0 ? paramFiles.toArray(type2) : null);

        return new BufferedReader(new StringReader(writer.toString()));

    }

    /**
  * adapted original code from file to apply the preprocessor
  * @param filePath relative path of the pig script
  * @param params parameters to be applied by the preprocessor
  * @throws IOException
  */
    public void registerScript(String filePath, List<String> params) throws IOException {
        try {
            InputStream stream = new FileInputStream(filePath);
            try {
             GruntParser grunt = new GruntParser(runParamPreprocessor(stream, params, new ArrayList<String>(), filePath));
             grunt.setInteractive(false);
             grunt.setParams(this);
             grunt.parseStopOnError(true);
            } finally {
             try {
              stream.close();
             } catch (IOException e) {
              e.printStackTrace();
             }
            }
        } catch (org.apache.pig.tools.pigscript.parser.ParseException e) {
            throw new IOException("Error while parsing script: "+filePath,e);
        } catch (org.apache.pig.tools.parameters.ParseException e) {
         throw new IOException("Error while parsing parameters: "+params,e);
  }
    }

    /**
  * adapted original code from file to apply the preprocessor
  * @param filePath relative path of the pig script
  * @param params parameters to be applied by the preprocessor
  * @throws IOException
  */
    public void registerScript(String filePath, String... params) throws IOException {
     registerScript(filePath, Arrays.asList(params));
    }

On 10/7/10 1:27 PM, "Olga Natkovich" <[EMAIL PROTECTED]> wrote:

Not at this point as parameter substitution is implemented a s preprocessor on the script.

Olga

-----Original Message-----
From: rakesh kothari [mailto:[EMAIL PROTECTED]]
Sent: Thursday, October 07, 2010 11:47 AM
To: [EMAIL PROTECTED]
Subject: Passing parameters to Pig Script using Java
Hi,

I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

Thanks,
-Rakesh