Hi Kurt,
>From the stack trace, it looks like it runs into an error while
estimating the size of the input.
Are all of the paths it's looking for there in hdfs:///user/kurt ?
does it work with pig_11 ? add --pig_version pig_11 to the oink command
also send out the command line you are using.
Thanks,
Julien

On Thu, Nov 15, 2012 at 9:15 AM, Kurt Smith <[EMAIL PROTECTED]> wrote:
> I'm getting this error when doing a manual run of the search_simplified
> twadoop query. This query has run fine before. Any idea what the issue is?
>
> Pig Stack Trace
> ---------------
> ERROR 2017: Internal error creating job configuration.
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
> ERROR 2017: Internal error creating job configuration.
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:738)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:264)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:150)
>         at org.apache.pig.PigServer.launchPlan(PigServer.java:1267)
>         at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1252)
>         at org.apache.pig.PigServer.execute(PigServer.java:1242)
>         at org.apache.pig.PigServer.executeBatch(PigServer.java:356)
>         at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:132)
>         at
> org.apache.pig.tools.grunt.GruntParser.processScript(GruntParser.java:452)
>         at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Script(PigScriptParser.java:752)
>         at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:423)
>         at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
>         at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
>         at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>         at org.apache.pig.Main.run(Main.java:561)
>         at org.apache.pig.Main.main(Main.java:111)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:971)
>         at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:944)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getInputSize(JobControlCompiler.java:840)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.estimateNumberOfReducers(JobControlCompiler.java:810)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:750)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:396)
>         ... 20 more
> ===============================================================================>
>
> from the output:
> ---
>
> 2012-11-15 07:54:02,554 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to
> the job
> 2012-11-15 07:54:02,555 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2012-11-15 07:54:03,005 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - BytesPerReducer=1610612736 maxReducers=999
> totalInputFileSize=1456250098769

Kurt, there's a lot of dependencies that this job expects at different
places, and if you run it as a normal user, it will get confused. Most
likely some input files are not found.

On 11/15/2012 09:36 AM, Julien Le Dem wrote:
> Hi Kurt,
>  From the stack trace, it looks like it runs into an error while
> estimating the size of the input.
> Are all of the paths it's looking for there in hdfs:///user/kurt ?
> does it work with pig_11 ? add --pig_version pig_11 to the oink command
> also send out the command line you are using.
> Thanks,
> Julien
>
> On Thu, Nov 15, 2012 at 9:15 AM, Kurt Smith <[EMAIL PROTECTED]> wrote:
>> I'm getting this error when doing a manual run of the search_simplified
>> twadoop query. This query has run fine before. Any idea what the issue is?
>>
>> Pig Stack Trace
>> ---------------
>> ERROR 2017: Internal error creating job configuration.
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
>> ERROR 2017: Internal error creating job configuration.
>>          at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:738)
>>          at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:264)
>>          at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:150)
>>          at org.apache.pig.PigServer.launchPlan(PigServer.java:1267)
>>          at
>> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1252)
>>          at org.apache.pig.PigServer.execute(PigServer.java:1242)
>>          at org.apache.pig.PigServer.executeBatch(PigServer.java:356)
>>          at
>> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:132)
>>          at
>> org.apache.pig.tools.grunt.GruntParser.processScript(GruntParser.java:452)
>>          at
>> org.apache.pig.tools.pigscript.parser.PigScriptParser.Script(PigScriptParser.java:752)
>>          at
>> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:423)
>>          at
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
>>          at
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
>>          at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>>          at org.apache.pig.Main.run(Main.java:561)
>>          at org.apache.pig.Main.main(Main.java:111)
>>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>          at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>          at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>          at java.lang.reflect.Method.invoke(Method.java:597)
>>          at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>> Caused by: java.lang.NullPointerException
>>          at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:971)
>>          at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:944)
>>          at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getInputSize(JobControlCompiler.java:840)
>>          at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.estimateNumberOfReducers(JobControlCompiler.java:810)
>>          at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:750)
>>          at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:396)
>>          ... 20 more
>> ===============================================================================>>
>>
>> from the output:
>> ---
>>
>> 2012-11-15 07:54:02,554 [main] INFO
>> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to
>> the job
>> 2012-11-15 07:54:02,555 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler

Aha, thanks Gabor. I'll just wait til Twadoop is building and let it fill
in.

On Thu, Nov 15, 2012 at 9:53 AM, Gabor Szabo <[EMAIL PROTECTED]> wrote:

> Kurt, there's a lot of dependencies that this job expects at different
> places, and if you run it as a normal user, it will get confused. Most
> likely some input files are not found.
>
>
> On 11/15/2012 09:36 AM, Julien Le Dem wrote:
>
>> Hi Kurt,
>>  From the stack trace, it looks like it runs into an error while
>> estimating the size of the input.
>> Are all of the paths it's looking for there in hdfs:///user/kurt ?
>> does it work with pig_11 ? add --pig_version pig_11 to the oink command
>> also send out the command line you are using.
>> Thanks,
>> Julien
>>
>> On Thu, Nov 15, 2012 at 9:15 AM, Kurt Smith <[EMAIL PROTECTED]> wrote:
>>
>>> I'm getting this error when doing a manual run of the search_simplified
>>> twadoop query. This query has run fine before. Any idea what the issue
>>> is?
>>>
>>> Pig Stack Trace
>>> ---------------
>>> ERROR 2017: Internal error creating job configuration.
>>>
>>> org.apache.pig.backend.hadoop.**executionengine.**mapReduceLayer.**
>>> JobCreationException:
>>> ERROR 2017: Internal error creating job configuration.
>>>          at
>>> org.apache.pig.backend.hadoop.**executionengine.**mapReduceLayer.**
>>> JobControlCompiler.getJob(**JobControlCompiler.java:738)
>>>          at
>>> org.apache.pig.backend.hadoop.**executionengine.**mapReduceLayer.**
>>> JobControlCompiler.compile(**JobControlCompiler.java:264)
>>>          at
>>> org.apache.pig.backend.hadoop.**executionengine.**mapReduceLayer.**
>>> MapReduceLauncher.launchPig(**MapReduceLauncher.java:150)
>>>          at org.apache.pig.PigServer.**launchPlan(PigServer.java:**1267)
>>>          at
>>> org.apache.pig.PigServer.**executeCompiledLogicalPlan(**
>>> PigServer.java:1252)
>>>          at org.apache.pig.PigServer.**execute(PigServer.java:1242)
>>>          at org.apache.pig.PigServer.**executeBatch(PigServer.java:**
>>> 356)
>>>          at
>>> org.apache.pig.tools.grunt.**GruntParser.executeBatch(**
>>> GruntParser.java:132)
>>>          at
>>> org.apache.pig.tools.grunt.**GruntParser.processScript(**
>>> GruntParser.java:452)
>>>          at
>>> org.apache.pig.tools.**pigscript.parser.**PigScriptParser.Script(**
>>> PigScriptParser.java:752)
>>>          at
>>> org.apache.pig.tools.**pigscript.parser.**PigScriptParser.parse(**
>>> PigScriptParser.java:423)
>>>          at
>>> org.apache.pig.tools.grunt.**GruntParser.parseStopOnError(**
>>> GruntParser.java:189)
>>>          at
>>> org.apache.pig.tools.grunt.**GruntParser.parseStopOnError(**
>>> GruntParser.java:165)
>>>          at org.apache.pig.tools.grunt.**Grunt.exec(Grunt.java:84)
>>>          at org.apache.pig.Main.run(Main.**java:561)
>>>          at org.apache.pig.Main.main(Main.**java:111)
>>>          at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native
>>> Method)
>>>          at
>>> sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>>> NativeMethodAccessorImpl.java:**39)
>>>          at
>>> sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>>> DelegatingMethodAccessorImpl.**java:25)
>>>          at java.lang.reflect.Method.**invoke(Method.java:597)
>>>          at org.apache.hadoop.util.RunJar.**main(RunJar.java:186)
>>> Caused by: java.lang.NullPointerException
>>>          at org.apache.hadoop.fs.**FileSystem.globStatus(**
>>> FileSystem.java:971)
>>>          at org.apache.hadoop.fs.**FileSystem.globStatus(**
>>> FileSystem.java:944)
>>>          at
>>> org.apache.pig.backend.hadoop.**executionengine.**mapReduceLayer.**
>>> JobControlCompiler.**getInputSize(**JobControlCompiler.java:840)
>>>          at
>>> org.apache.pig.backend.hadoop.**executionengine.**mapReduceLayer.**
>>> JobControlCompiler.**estimateNumberOfReducers(**
>>> JobControlCompiler.java:810)
>>>          at
>>> org.apache.pig.backend.hadoop.**executionengine.**mapReduceLayer.**
>>> JobControlCompiler.**adjustNumReducers(**JobControlCompiler.java:750)
Kurt Smith
Senior Data Scientist, Analytics | Twitter, Inc
@kurtosis0 <https://twitter.com/intent/user?screen_name=kurtosis0>