Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> AvroStorage schema_uri pointing to local file doesn't work


Copy link to this message
-
AvroStorage schema_uri pointing to local file doesn't work
Hey guys,

I am using AvroStorage like this:

STORE alias INTO '$OUTPUT'
    USING org.apache.pig.piggybank.storage.avro.AvroStorage('{
    "index" : 1,
    "schema_uri": "file://path/schema.avsc"}');

so, it is explicit to take the schema.avsc from the local file system, not
HDFS.
It works in a pseudo-distributed cluster, but fails on a normal cluster
with java.io.FileNotFoundException for the schema file
Looks like this is happening in backend.
I assume this is because the backend invocation of AvroStorage on a node,
different from the node I am running the pig script from, cannot find the
file in the local file system.
Why can't it use the schema file from front-end invocation?
Does it mean that I am only limited to either HDFS locations for schema_uri
or using embedding the schema string in AvroStorage parameters?

Thanks in advance

Ruslan Al-Fakikh
+
Cheolsoo Park 2013-12-25, 07:48
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB