Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Pig Schema contains a name that is not allowed in Avro


Copy link to this message
-
Pig Schema contains a name that is not allowed in Avro
Hi!

I'm getting the following error when running the script below in grunt
(pig 0.12.0):

DEFINE AvroStorageSTORE AvroStorage('-f schema.avsc');
data = LOAD '/path/to/data' AS(a:long, b:long, c:int, d:chararray,
e:int, f:int, g:int, h:long, i:chararray, j:int, k:int, l:int, m:int,
n:int, o:int, p:int, q:int, r:int, s:int, t:int, u:int, v:chararray,
w:chararray, x:chararray);
STORE data into '/path/to/output' USING AvroStorageSTORE();

2013-10-21 18:50:15,491 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2013-10-21 18:50:15,554 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2116:
<line 3, column 0> Output Location Validation Failed for:
'hdfs://path/to/output More info to follow:
Pig Schema contains a name that is not allowed in Avro
Details at logfile: pig_1382374188771.log

Logfile contains:

Pig Schema contains a name that is not allowed in Avro

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
to store alias data
        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1596)
        at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
        at
org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
        at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
        at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
        at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
        at org.apache.pig.Main.run(Main.java:495)
        at org.apache.pig.Main.main(Main.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 2116:
<line 3, column 0> Output Location Validation Failed for:
'hdfs://path/to/output More info to follow:
Pig Schema contains a name that is not allowed in Avro
        at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
        at
org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
        at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
        at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
        at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
        at
org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
        at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
        at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
        at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:294)
        at org.apache.pig.PigServer.compilePp(PigServer.java:1360)
        at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297)
        at org.apache.pig.PigServer.execute(PigServer.java:1289)
        at org.apache.pig.PigServer.access$400(PigServer.java:125)
        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1591)
        ... 13 more
Caused by: java.io.IOException: Pig Schema contains a name that is not
allowed in Avro
        at
org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.toAvroName(AvroStorageSchemaConversionUtilities.java:380)
        at
org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:326)
        at
org.apache.pig.builtin.AvroStorage.checkSchema(AvroStorage.java:485)
        at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65)
        ... 26 more
Avro schema looks like:

{
"name": "entry",
"namespace": "a.b.c",
"type": "record",
"doc": "A Log",
"fields": [
{"name": "a", "type": "long"},
{"name": "b", "type": "long"},
{"name": "c" "type": "int"},
{"name": "d" "type": {
"name": "d"
"type": "enum",
"symbols": [
"A",
"B",
"C",
"D",
"E"
],
"doc": "Some doc"
}
},
{"name": "e" "type": "int"},
{"name": "f", "type": "int"},
{"name": "g", "type": "int"},
{"name": "h", "type": ["null", "long"], "default": null},
{"name": "i", "type": ["null", "string"], "default": null},
{"name": "j", "type": "int"},
{"name": "k", "type": "int"},
{"name": "l", "type": "int"},
{"name": "m", "type": ["null", "int"], "default": null},
{"name": "n", "type": ["null", "int"], "default": null},
{"name": "o", "type": ["null", "int"], "default": null},
{"name": "p", "type": ["null", "int"], "default": null},
{"name": "q", "type": ["null", "int"], "default": null},
{"name": "r", "type": ["null", "int"], "default": null},
{"name": "s", "type": ["null", "int"], "default": null},
{"name": "t", "type": "boolean"},
{"name": "u", "type": "int"},
{"name": "v", "type": "string"},
{"name": "w", "type": ["null", "string"], "default": null},
{"name": "x", "type": "string"}
]
}
Had to redact the names because of confidentiality.

Do you have any idea what the problem could be? The error message seems
misleading...

Greetings,

Johannes Schwenk

Softwareentwickler (Reporting)
________________________________________________________

ADITION technologies AG
Schwarzwaldstraße 78b
79117 Freiburg

http://www.adition.com

T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
SUPPORT +49  / (0)1805 - ADITION

(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)

Eingetragen beim Amtsgericht Düsseldorf unter HRB
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB