Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Pig Schema contains a name that is not allowed in Avro


Copy link to this message
-
Pig Schema contains a name that is not allowed in Avro
Hi!

I'm getting the following error when running the script below in grunt
(pig 0.12.0):

DEFINE AvroStorageSTORE AvroStorage('-f schema.avsc');
data = LOAD '/path/to/data' AS(a:long, b:long, c:int, d:chararray,
e:int, f:int, g:int, h:long, i:chararray, j:int, k:int, l:int, m:int,
n:int, o:int, p:int, q:int, r:int, s:int, t:int, u:int, v:chararray,
w:chararray, x:chararray);
STORE data into '/path/to/output' USING AvroStorageSTORE();

2013-10-21 18:50:15,491 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2013-10-21 18:50:15,554 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2116:
<line 3, column 0> Output Location Validation Failed for:
'hdfs://path/to/output More info to follow:
Pig Schema contains a name that is not allowed in Avro
Details at logfile: pig_1382374188771.log

Logfile contains:

Pig Schema contains a name that is not allowed in Avro

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
to store alias data
        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1596)
        at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
        at
org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967)
        at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
        at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
        at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
        at org.apache.pig.Main.run(Main.java:495)
        at org.apache.pig.Main.main(Main.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 2116:
<line 3, column 0> Output Location Validation Failed for:
'hdfs://path/to/output More info to follow:
Pig Schema contains a name that is not allowed in Avro
        at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
        at
org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
        at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
        at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
        at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
        at
org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
        at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
        at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
        at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:294)
        at org.apache.pig.PigServer.compilePp(PigServer.java:1360)
        at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297)
        at org.apache.pig.PigServer.execute(PigServer.java:1289)
        at org.apache.pig.PigServer.access$400(PigServer.java:125)
        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1591)
        ... 13 more
Caused by: java.io.IOException: Pig Schema contains a name that is not
allowed in Avro
        at
org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.toAvroName(AvroStorageSchemaConversionUtilities.java:380)
        at
org.apache.pig.impl.util.avro.AvroStorageSchemaConversionUtilities.resourceSchemaToAvroSchema(AvroStorageSchemaConversionUtilities.java:326)
        at
org.apache.pig.builtin.AvroStorage.checkSchema(AvroStorage.java:485)
        at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65)
        ... 26 more
Avro schema looks like:

{
"name": "entry",
"namespace": "a.b.c",
"type": "record",
"doc": "A Log",
"fields": [
{"name": "a", "type": "long"},
{"name": "b", "type": "long"},
{"name": "c" "type": "int"},
{"name": "d" "type": {
"name": "d"
"type": "enum",
"symbols": [
"A",
"B",
"C",
"D",
"E"
],
"doc": "Some doc"
}
},
{"name": "e" "type": "int"},
{"name": "f", "type": "int"},
{"name": "g", "type": "int"},
{"name": "h", "type": ["null", "long"], "default": null},
{"name": "i", "type": ["null", "string"], "default": null},
{"name": "j", "type": "int"},
{"name": "k", "type": "int"},
{"name": "l", "type": "int"},
{"name": "m", "type": ["null", "int"], "default": null},
{"name": "n", "type": ["null", "int"], "default": null},
{"name": "o", "type": ["null", "int"], "default": null},
{"name": "p", "type": ["null", "int"], "default": null},
{"name": "q", "type": ["null", "int"], "default": null},
{"name": "r", "type": ["null", "int"], "default": null},
{"name": "s", "type": ["null", "int"], "default": null},
{"name": "t", "type": "boolean"},
{"name": "u", "type": "int"},
{"name": "v", "type": "string"},
{"name": "w", "type": ["null", "string"], "default": null},
{"name": "x", "type": "string"}
]
}
Had to redact the names because of confidentiality.

Do you have any idea what the problem could be? The error message seems
misleading...

Greetings,

Johannes Schwenk

Softwareentwickler (Reporting)
________________________________________________________

ADITION technologies AG
Schwarzwaldstraße 78b
79117 Freiburg

http://www.adition.com

T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
SUPPORT +49  / (0)1805 - ADITION

(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)

Eingetragen beim Amtsgericht Düsseldorf unter HRB
+
j.barrett Strausser 2013-10-21, 17:29
+
Johannes Schwenk 2013-10-22, 08:16
+
Ruslan Al-Fakikh 2013-11-19, 16:40