Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - AvroStorage load and store, schema with maps


Copy link to this message
-
AvroStorage load and store, schema with maps
Johannes Schwenk 2012-08-23, 15:49
Hi all,

I'm trying to execute the following pig script with pig-0.10.0 and yarn
(cdh4.0.0):

--
DEFINE AvroStorage org.apache.pig.piggybank.storage.avro.AvroStorage();
loaded_data = LOAD '$input' USING AvroStorage();
STORE loaded_data INTO '$output' USING AvroStorage('same', '$input');
--

I call the pig this way:

pig
-Dpig.additional.jars=lib/piggybank.jar:lib/json-simple-1.1.jar:lib/avro-1.5.3.jar
-file script.pig -param input=input.avro -param output=output.avro

The input.avro has the following schema:

http://pastebin.com/ZWU6qLWx

I always get

<file script.pig, line 3, column 0> Output Location Validation Failed
for: 'xxx/output.avro' More info to follow:
Please provide schema for Map field!
Details at logfile: xxx/pig_1345735999390.log

Log excerpt:

Please provide schema for Map field!
        at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
        at
org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
        at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
        at
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
        at
org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
        at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
        at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
        at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:293)
        at org.apache.pig.PigServer.compilePp(PigServer.java:1316)
        at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1253)
        at org.apache.pig.PigServer.execute(PigServer.java:1245)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:362)
        at
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:132)
        at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
        at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
        at org.apache.pig.Main.run(Main.java:430)
        at org.apache.pig.Main.main(Main.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.io.IOException: Please provide schema for Map field!
        at
org.apache.pig.piggybank.storage.avro.PigSchema2Avro.convert(PigSchema2Avro.java:110)
        at
org.apache.pig.piggybank.storage.avro.PigSchema2Avro.convertRecord(PigSchema2Avro.java:151)
        at
org.apache.pig.piggybank.storage.avro.PigSchema2Avro.convert(PigSchema2Avro.java:62)
        at
org.apache.pig.piggybank.storage.avro.AvroStorage.checkSchema(AvroStorage.java:534)
        at
org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:65)
        ... 22 more
I also tried to specify

AvroStorage('{"debug": 5, "schema_file": "schema.avsc", "field22",
"def:pd", "field23", "def:epd"}')

- same result.
Do you have any hints?

Greetings,
Johannes Schwenk

--
Softwareentwickler (Reporting)
________________________________________________________

ADITION technologies AG
Schwarzwaldstraße 78b
79117 Freiburg

http://www.adition.com

T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
SUPPORT +49  / (0)1805 - ADITION

(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)

Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
UStIDNr.: DE 218 858 434
+
Cheolsoo Park 2012-08-23, 18:11
+
Cheolsoo Park 2012-08-23, 19:02
+
Johannes Schwenk 2012-09-03, 11:16