Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> JsonStorage() fails to write .pig_schema to S3 after correctly writing alias to json files.


Copy link to this message
-
Re: JsonStorage() fails to write .pig_schema to S3 after correctly writing alias to json files.
I'm using the builtin JsonStorage() AFAIK (e.g. the one documented in the
Pig manual).
On Sat, Jun 8, 2013 at 7:14 PM, Shahab Yunus <[EMAIL PROTECTED]> wrote:

> I don't have a solution but just wondering what lib are you using for
> JsonStorage? I hope it doesn't have any issues in it.
>
> Regards,
> Shahab
>
>
> On Sat, Jun 8, 2013 at 3:26 PM, Alan Crosswell <[EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > I've been having trouble with JsonStorage(). First, since my Python UDF
> had
> > an outputSchema that returned floats, I was getting an error in
> JsonStorage
> > trying to cast Double to Float. I resolved this by changing my UDF to
> > return doubles.
> >
> > Pig-0.11.1, hadoop-1.0.3.
> >
> > Next, I am able to successfully write json files out to s3 (I was
> watching
> > as my Pig job was running and grabbed a sample) but then at what appears
> to
> > be the final step of writing .pig_schema, this error is thrown:
> >
> > grunt> *STORE firsts INTO 's3n://n2ygk/firsthops.json' using
> > JsonStorage();*
> > *
> > *
> > *... *chugs along for a while successfully writing
> > s3://n2ygk/firsthops.json/part-r-* into the bucket.... and then:
> >
> > *java.lang.IllegalArgumentException: This file system object (hdfs://
> > 10.253.44.244:9000) does not support access to the request path
> > 's3n://n2ygk/firsthops.json/.pig_schema' You possibly called
> > FileSystem.get(conf) when you should have called FileSystem.get(uri,
> conf)
> > to obtain a file system supporting your path.*
> >                                     at
> > org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:384)
> >                                     at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:129)
> >                                     at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:513)
> >                                     at
> > org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:770)
> >                                     at
> >
> >
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200)
> >                                     at
> >
> >
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128)
> >                                     at
> >
> >
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:144)
> >                                     at
> > org.apache.pig.builtin.JsonMetadata.storeSchema(JsonMetadata.java:294)
> >                                     at
> > org.apache.pig.builtin.JsonStorage.storeSchema(JsonStorage.java:274)
> >                                     at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.storeCleanup(PigOutputCommitter.java:141)
> >                                     at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:204)
> >                                     at
> > org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:1060)
> >                                     at
> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:362)
> >                                     at
> > org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> >                                     at
> > java.security.AccessController.doPrivileged(Native Method)
> >                                     at
> > javax.security.auth.Subject.doAs(Subject.java:396)
> >                                     at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
> >                                     at
> > org.apache.hadoop.mapred.Child.main(Child.java:249)
> >
> > Any ideas?
> >
> > Thanks.
> > /a
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB