Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Problems loading a datafile..


Copy link to this message
-
Re: Problems loading a datafile..
Kris Coward 2011-03-02, 17:25

Nope; they're reproduced across all the machines. Does the
LzoTokenizedLoader class have any dependencies that LzoTokenizedStorage
doesn't (which I may be overlooking)?

-K

On Tue, Mar 01, 2011 at 07:17:10PM -0500, Kris Coward wrote:
>
> What's peculiar is that the test script for the loader class that was
> run a week ago seems also to be failing with the same error. We've added
> nodes to the cluster; maybe the relevant .jar files haven't been copied
> over to those nodes. I'll bug our sysadmin about that..
>
> Thanks,
> Kris
>
> On Tue, Mar 01, 2011 at 02:08:32PM -0800, Dmitriy Ryaboy wrote:
> > Kris,
> > Check the pig log file. Often "unable to create input slice" is caused by
> > errors such as not being able to find your loader class, or some dependency
> > of your loader class.
> >
> > D
> >
> > On Tue, Mar 1, 2011 at 1:48 PM, Kris Coward <[EMAIL PROTECTED]> wrote:
> >
> > >
> > > I get the output:
> > >
> > > rw-r--r--   2 kris supergroup     172694 2011-02-25 01:59
> > > /path/to/file/item/ex/subdir
> > >
> > > -K
> > >
> > > On Tue, Mar 01, 2011 at 12:46:31PM -0800, Dmitriy Ryaboy wrote:
> > > > What happens when you "hadoop fs -lsr" those paths?
> > > >
> > > > D
> > > >
> > > > On Sun, Feb 27, 2011 at 7:47 PM, Kris Coward <[EMAIL PROTECTED]> wrote:
> > > >
> > > > >
> > > > > So I finally got a couple of test scripts running on my cluster to take
> > > > > a sample data file, load it, do a little processing, store it, load it,
> > > > > do a little more processing, and dump the results.
> > > > >
> > > > > Once these were working, I set to parsing and storing some real data,
> > > > > but when got an "Unable to create input slice" error when trying to
> > > load
> > > > > this data back out again. This happened with each of:
> > > > >
> > > > > foo = LOAD '/path/to/file/{item,list,glob}/*/subdir' USING
> > > > > com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS
> > > (schema:...);
> > > > > foo = LOAD '/path/to/file/item/*/subdir' USING
> > > > > com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS
> > > (schema:...);
> > > > > foo = LOAD '/path/to/file/item/ex/subdir' USING
> > > > > com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS
> > > (schema:...);
> > > > >
> > > > > and yielded the error (the same each time, except for the name/glob
> > > > > used):
> > > > >
> > > > > ERROR 2997: Unable to recreate exception from backend error:
> > > > > org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> > > Unable to
> > > > > create input slice for:
> > > > > hdfs://master.hadoop:9000//path/to/file/item/ex/subdir
> > > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
> > > to
> > > > > open iterator for alias foo
> > > > >        at org.apache.pig.PigServer.openIterator(PigServer.java:482)
> > > > >        at
> > > > >
> > > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
> > > > >        at
> > > > >
> > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
> > > > >        at
> > > > >
> > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
> > > > >        at
> > > > >
> > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
> > > > >        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
> > > > >        at org.apache.pig.Main.main(Main.java:352)
> > > > > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
> > > > > 2997: Unable to recreate exception from backend error:
> > > > > org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> > > Unable to
> > > > > create input slice for:
> > > > > hdfs://master.hadoop:9000/path/to/file/item/ex/subdir
> > > > >        at
> > > > >
> > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:176)
> > > > >        at
> > > > >
> > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:253)

Kris Coward http://unripe.melon.org/
GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB 12B3