Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - "Failed to create DataStorage" ?


Copy link to this message
-
Re: "Failed to create DataStorage" ?
Prashant Kommireddi 2013-03-21, 23:17
It has to be a classpath issue with a different hadoop version being picked
up. I can very easily reproduce the same issue on my end (copied at the end
of this email)

Regarding your other question, running pig from command-line will let you
run UDFs and provide most functionality you would have otherwise. Note that
complex looping sometimes requires java/python or other pig supported
languages. If you don't have such a requirement command-line will work just
fine (it is also what's generally used more than Pig's Java APIs).

I would recommend trying to check on the classpath for your issue though in
any case.

Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Failed to create DataStorage

java.lang.RuntimeException: Failed to create DataStorage
    at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:205)
    at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:118)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:208)
    at org.apache.pig.PigServer.<init>(PigServer.java:246)
    at org.apache.pig.PigServer.<init>(PigServer.java:231)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47)
    at org.apache.pig.Main.run(Main.java:487)
    at org.apache.pig.Main.main(Main.java:111)
Caused by: java.io.IOException: Call to x.y.z.net/XX.XX.XX.XX:54310 failed
on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
    at org.apache.hadoop.ipc.Client.call(Client.java:1071)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at $Proxy1.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
    at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:800)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:745)
===============================================================================

On Thu, Mar 21, 2013 at 2:15 PM, Ryan Compton <[EMAIL PROTECTED]>wrote:

> java -cp
> target/mybigjar-1.1-SNAPSHOT-jar-with-dependencies.jar:$HADOOP_CONF_DIR:/home/rfcompton/Downloads/pig-0.11.0-src/pig-withouthadoop.jar
> org.apache.pig.Main
>
> Still can't create DataStorage. How far will I be able to get with pig
> if I restrict myself to the command line? UDFs will still work, right?
>
> On Thu, Mar 21, 2013 at 1:54 PM, Prashant Kommireddi
> <[EMAIL PROTECTED]> wrote:
> > Apologies, just read your email again to realize your Java program is
> > what's not working. Your command-line setup seems fine.
> >
> > Can you try running your program with
> > /home/rfcompton/Downloads/pig-0.11.0-withouthadoop.jar instead of
> > /home/rfcompton/Downloads/pig-0.11.0-src/pig.jar and make sure 0.20.2
> > hadoop is on the classpath.
> >
> >
> > On Thu, Mar 21, 2013 at 1:36 PM, Ryan Compton <[EMAIL PROTECTED]
> >wrote: