Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> SchemaTuple doesn't seem to work on YARN


Copy link to this message
-
Re: SchemaTuple doesn't seem to work on YARN
Hello!

I implemented the SchemaTuple stuff. Glad to hear you're trying it out! I
did not test it with YARN at all. It looks like the way that the filesystem
and distributed cache work have changed. I myself am not super up on that,
but perhaps there is known documentation on how it differs? The way that it
works is that when the pig script is processed, a bunch of code is
generated and added to the distributed cache. Then, each mapper and reducer
copies that code locally, so that a UrlClassLoader can load it. Any piece
of that could have changed...
2013/9/3 Matt Christiansen <[EMAIL PROTECTED]>

> Hello; we are running a YARN cluster and I went to go try out the
> pig.schematuple option on our cluster (yes I know its experimental)
> and it doesn't seem to work the jobs all error out with:
>
>
> 2013-09-03 15:20:49,732 INFO [main]
> org.apache.pig.data.SchemaTupleBackend: Copying files in key
> [pig.schematuple.classes] from distributed cache:
>
> SchemaTuple_14$1.class,SchemaTuple_63.class,SchemaTuple_30.class,SchemaTuple_9$1.class,SchemaTuple_36.class,SchemaTuple_5$1.class,SchemaTuple_49.class,SchemaTuple_47.class,SchemaTuple_8.class,SchemaTuple_23.class,SchemaTuple_19$1.class,SchemaTuple_32.class,SchemaTuple_11.class,SchemaTuple_71.class,SchemaTuple_23$1.class,SchemaTuple_73.class,SchemaTuple_1$1.class,SchemaTuple_58.class,SchemaTuple_65.class,SchemaTuple_75$1.class,SchemaTuple_26.class,SchemaTuple_6.class,SchemaTuple_10$1.class,SchemaTuple_1.class,SchemaTuple_58$1.class,SchemaTuple_51$1.class,SchemaTuple_3$1.class,SchemaTuple_67$1.class,SchemaTuple_35.class,SchemaTuple_38.class,SchemaTuple_11$1.class,SchemaTuple_74$1.class,SchemaTuple_45$1.class,SchemaTuple_12$1.class,SchemaTuple_0$1.class,SchemaTuple_10.class,SchemaTuple_14.class,SchemaTuple_76$1.class,SchemaTuple_49$1.class,SchemaTuple_30$1.class,SchemaTuple_56.class,SchemaTuple_48$1.class,SchemaTuple_43$1.class,SchemaTuple_16$1.class,SchemaTuple_61$1.class,SchemaTuple_15$1.class,SchemaTuple_21$1.class,SchemaTuple_59$1.class,SchemaTuple_76.class,SchemaTuple_50$1.class,SchemaTuple_64$1.class,SchemaTuple_44$1.class,SchemaTuple_72.class,SchemaTuple_51.class,SchemaTuple_25$1.class,SchemaTuple_43.class,SchemaTuple_57.class,SchemaTuple_62$1.class,SchemaTuple_16.class,SchemaTuple_66.class,SchemaTuple_57$1.class,SchemaTuple_20$1.class,SchemaTuple_33.class,SchemaTuple_21.class,SchemaTuple_68.class,SchemaTuple_74.class,SchemaTuple_48.class,SchemaTuple_54$1.class,SchemaTuple_19.class,SchemaTuple_71$1.class,SchemaTuple_38$1.class,SchemaTuple_42.class,SchemaTuple_18.class,SchemaTuple_37$1.class,SchemaTuple_39$1.class,SchemaTuple_64.class,SchemaTuple_41$1.class,SchemaTuple_52$1.class,SchemaTuple_7.class,SchemaTuple_28$1.class,SchemaTuple_13.class,SchemaTuple_69.class,SchemaTuple_72$1.class,SchemaTuple_41.class,SchemaTuple_56$1.class,SchemaTuple_0.class,SchemaTuple_53.class,SchemaTuple_60.class,SchemaTuple_40.class,SchemaTuple_66$1.class,SchemaTuple_24$1.class,SchemaTuple_60$1.class,SchemaTuple_2$1.class,SchemaTuple_47$1.class,SchemaTuple_28.class,SchemaTuple_68$1.class,SchemaTuple_39.class,SchemaTuple_6$1.class,SchemaTuple_69$1.class,SchemaTuple_50.class,SchemaTuple_40$1.class,SchemaTuple_62.class,SchemaTuple_31$1.class,SchemaTuple_46$1.class,SchemaTuple_20.class,SchemaTuple_13$1.class,SchemaTuple_37.class,SchemaTuple_24.class,SchemaTuple_8$1.class,SchemaTuple_9.class,SchemaTuple_22$1.class,SchemaTuple_46.class,SchemaTuple_65$1.class,SchemaTuple_29.class,SchemaTuple_22.class,SchemaTuple_29$1.class,SchemaTuple_67.class,SchemaTuple_45.class,SchemaTuple_44.class,SchemaTuple_4$1.class,SchemaTuple_18$1.class,SchemaTuple_12.class,SchemaTuple_17$1.class,SchemaTuple_34.class,SchemaTuple_53$1.class,SchemaTuple_35$1.class,SchemaTuple_75.class,SchemaTuple_31.class,SchemaTuple_70.class,SchemaTuple_7$1.class,SchemaTuple_32$1.class,SchemaTuple_17.class,SchemaTuple_5.class,SchemaTuple_61.class,SchemaTuple_25.class,SchemaTuple_63$1.class,SchemaTuple_70$1.class,SchemaTuple_55$1.class,SchemaTuple_55.class,SchemaTuple_4.class,SchemaTuple_34$1.class,SchemaTuple_59.class,SchemaTuple_36$1.class,SchemaTuple_2.class,SchemaTuple_3.class,SchemaTuple_54.class,SchemaTuple_73$1.class,SchemaTuple_27$1.class,SchemaTuple_27.class,SchemaTuple_15.class,SchemaTuple_26$1.class,SchemaTuple_33$1.class,SchemaTuple_52.class,SchemaTuple_42$1.class