Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> AvroStorage schema_uri pointing to local file doesn't work


Copy link to this message
-
AvroStorage schema_uri pointing to local file doesn't work
Hey guys,

I am using AvroStorage like this:

STORE alias INTO '$OUTPUT'
    USING org.apache.pig.piggybank.storage.avro.AvroStorage('{
    "index" : 1,
    "schema_uri": "file://path/schema.avsc"}');

so, it is explicit to take the schema.avsc from the local file system, not
HDFS.
It works in a pseudo-distributed cluster, but fails on a normal cluster
with java.io.FileNotFoundException for the schema file
Looks like this is happening in backend.
I assume this is because the backend invocation of AvroStorage on a node,
different from the node I am running the pig script from, cannot find the
file in the local file system.
Why can't it use the schema file from front-end invocation?
Does it mean that I am only limited to either HDFS locations for schema_uri
or using embedding the schema string in AvroStorage parameters?

Thanks in advance

Ruslan Al-Fakikh
+
Cheolsoo Park 2013-12-25, 07:48