Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - Re: [tw-hadoop-users] ERROR 2017: Internal error creating job configuration.


Copy link to this message
-
Re: [tw-hadoop-users] ERROR 2017: Internal error creating job configuration.
Julien Le Dem 2012-11-15, 17:36
Hi Kurt,
>From the stack trace, it looks like it runs into an error while
estimating the size of the input.
Are all of the paths it's looking for there in hdfs:///user/kurt ?
does it work with pig_11 ? add --pig_version pig_11 to the oink command
also send out the command line you are using.
Thanks,
Julien

On Thu, Nov 15, 2012 at 9:15 AM, Kurt Smith <[EMAIL PROTECTED]> wrote:
> I'm getting this error when doing a manual run of the search_simplified
> twadoop query. This query has run fine before. Any idea what the issue is?
>
> Pig Stack Trace
> ---------------
> ERROR 2017: Internal error creating job configuration.
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
> ERROR 2017: Internal error creating job configuration.
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:738)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:264)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:150)
>         at org.apache.pig.PigServer.launchPlan(PigServer.java:1267)
>         at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1252)
>         at org.apache.pig.PigServer.execute(PigServer.java:1242)
>         at org.apache.pig.PigServer.executeBatch(PigServer.java:356)
>         at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:132)
>         at
> org.apache.pig.tools.grunt.GruntParser.processScript(GruntParser.java:452)
>         at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.Script(PigScriptParser.java:752)
>         at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:423)
>         at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
>         at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
>         at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>         at org.apache.pig.Main.run(Main.java:561)
>         at org.apache.pig.Main.main(Main.java:111)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:971)
>         at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:944)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getInputSize(JobControlCompiler.java:840)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.estimateNumberOfReducers(JobControlCompiler.java:810)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.adjustNumReducers(JobControlCompiler.java:750)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:396)
>         ... 20 more
> ===============================================================================>
>
> from the output:
> ---
>
> 2012-11-15 07:54:02,554 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to
> the job
> 2012-11-15 07:54:02,555 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2012-11-15 07:54:03,005 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - BytesPerReducer=1610612736 maxReducers=999
> totalInputFileSize=1456250098769
+
Gabor Szabo 2012-11-15, 17:53
+
Kurt Smith 2012-11-15, 17:57