Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Problems loading a datafile..


Copy link to this message
-
Re: Problems loading a datafile..

What's peculiar is that the test script for the loader class that was
run a week ago seems also to be failing with the same error. We've added
nodes to the cluster; maybe the relevant .jar files haven't been copied
over to those nodes. I'll bug our sysadmin about that..

Thanks,
Kris

On Tue, Mar 01, 2011 at 02:08:32PM -0800, Dmitriy Ryaboy wrote:
> Kris,
> Check the pig log file. Often "unable to create input slice" is caused by
> errors such as not being able to find your loader class, or some dependency
> of your loader class.
>
> D
>
> On Tue, Mar 1, 2011 at 1:48 PM, Kris Coward <[EMAIL PROTECTED]> wrote:
>
> >
> > I get the output:
> >
> > rw-r--r--   2 kris supergroup     172694 2011-02-25 01:59
> > /path/to/file/item/ex/subdir
> >
> > -K
> >
> > On Tue, Mar 01, 2011 at 12:46:31PM -0800, Dmitriy Ryaboy wrote:
> > > What happens when you "hadoop fs -lsr" those paths?
> > >
> > > D
> > >
> > > On Sun, Feb 27, 2011 at 7:47 PM, Kris Coward <[EMAIL PROTECTED]> wrote:
> > >
> > > >
> > > > So I finally got a couple of test scripts running on my cluster to take
> > > > a sample data file, load it, do a little processing, store it, load it,
> > > > do a little more processing, and dump the results.
> > > >
> > > > Once these were working, I set to parsing and storing some real data,
> > > > but when got an "Unable to create input slice" error when trying to
> > load
> > > > this data back out again. This happened with each of:
> > > >
> > > > foo = LOAD '/path/to/file/{item,list,glob}/*/subdir' USING
> > > > com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS
> > (schema:...);
> > > > foo = LOAD '/path/to/file/item/*/subdir' USING
> > > > com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS
> > (schema:...);
> > > > foo = LOAD '/path/to/file/item/ex/subdir' USING
> > > > com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS
> > (schema:...);
> > > >
> > > > and yielded the error (the same each time, except for the name/glob
> > > > used):
> > > >
> > > > ERROR 2997: Unable to recreate exception from backend error:
> > > > org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> > Unable to
> > > > create input slice for:
> > > > hdfs://master.hadoop:9000//path/to/file/item/ex/subdir
> > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
> > to
> > > > open iterator for alias foo
> > > >        at org.apache.pig.PigServer.openIterator(PigServer.java:482)
> > > >        at
> > > >
> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
> > > >        at
> > > >
> > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
> > > >        at
> > > >
> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
> > > >        at
> > > >
> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
> > > >        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
> > > >        at org.apache.pig.Main.main(Main.java:352)
> > > > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
> > > > 2997: Unable to recreate exception from backend error:
> > > > org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> > Unable to
> > > > create input slice for:
> > > > hdfs://master.hadoop:9000/path/to/file/item/ex/subdir
> > > >        at
> > > >
> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:176)
> > > >        at
> > > >
> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:253)
> > > >        at
> > > >
> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:249)
> > > >        at
> > > > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:781)
> > > >        at org.apache.pig.PigServer.store(PigServer.java:529)
> > > >        at org.apache.pig.PigServer.openIterator(PigServer.java:465)
> > > >        ... 6 more

Kris Coward http://unripe.melon.org/
GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB 12B3
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB