-
Re: Pig 0.10.0 and Hadoop 0.23Johannes Schwenk 2012-07-03, 16:12
Thanks.
Well, I did that and rebuild all pig jars with -Dhadoopversion=23 but am
still getting errors - this time around it's errors with opening an
iterator on alias b again, sigh...
I have singled out one of the similarly failing tests and pasted the output:
http://pastebin.com/reKjcX0p
The test case failing looks like this:
http://pastebin.com/L5seh0ka
And INPUT_FILE gets created correctly through @BeforeClass
http://pastebin.com/wKrH7i50
If I run the script in the grunt shell invoked like this
pig -x local -propertyFile
/home/schwenk/dev/hadoop/trunk/pig/conf/adition.properties
everything works fine and I can execute the script
grunt> a = LOAD 'input.dat' AS (id:int, grp:int, additional:int,
referer:chararray);
b = FILTER a BY com.adition.pig.filtering.string.CONTAINS(referer, 'obama');
grunt> dump b;
...
2012-07-03 18:11:23,032 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1
(4,323,242,http://www.google.com/url&url=http%3A%2F%2Fwww.tagesschau.de&q=barack%20obama)
(5,423,342,http://www.google.com/url&url=http%3A%2F%2Fwww.bild.de&q=obama)
and verify that the output is correct.
Any further advice on this?
Thanks,
Johannes
Am 27.06.2012 22:24, schrieb Daniel Dai:
> Do "ant clean".
>
> On Thu, Jun 21, 2012 at 1:16 AM, Johannes Schwenk <
> [EMAIL PROTECTED]> wrote:
>
>> Am 20.06.2012 20:21, schrieb Daniel Dai:
>>> Have you recompiled Pig using -Dhadoopversion=23?
>>
>> No, thanks that solved the issue at hand!
>>
>> I have other errors though. My test cases return errors like
>>
>> http://pastebin.com/XpszYeu2
>>
>> So I am still doing something wrong it seems?!
>>
>> Thanks,
>> Johannes
>>
>>
>>> On Wed, Jun 20, 2012 at 4:38 AM, Johannes Schwenk <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi list,
>>>>
>>>> I have written two UDFs that run fine with Hadoop 0.20.2 and Pig 0.10.0.
>>>> I am trying to switch from Hadoop 0.20.2 to 0.23 and am encountering
>>>> some errors on running my tests:
>>>>
>>>> http://pastebin.com/JwHRQ9RQ
>>>>
>>>> Might this be a simple error in configuration?
>>>>
>>>> Johannes Schwenk
>>>>
>>>> --
>>>> Softwareentwickler (Reporting)
>>>> ________________________________________________________
>>>>
>>>> ADITION technologies AG
>>>> Schwarzwaldstraße 78b
>>>> 79117 Freiburg
>>>>
>>>> http://www.adition.com
>>>>
>>>> T +49 / (0)761 / 88147 - 30
>>>> F +49 / (0)761 / 88147 - 77
>>>> SUPPORT +49 / (0)1805 - ADITION
>>>>
>>>> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>>>>
>>>> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
>>>> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus
>> Schlüter
>>>> Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
>>>> UStIDNr.: DE 218 858 434
>>>>
>>>>
>>>
>>
>>
>>
>> Johannes Schwenk
>>
>> --
>> Softwareentwickler (Reporting)
>> ________________________________________________________
>>
>> ADITION technologies AG
>> Schwarzwaldstraße 78b
>> 79117 Freiburg
>>
>> http://www.adition.com
>>
>> T +49 / (0)761 / 88147 - 30
>> F +49 / (0)761 / 88147 - 77
>> SUPPORT +49 / (0)1805 - ADITION
>>
>> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>>
>> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
>> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
>> Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
>> UStIDNr.: DE 218 858 434
>>
>>
>
Johannes Schwenk
--
Softwareentwickler (Reporting)
________________________________________________________
ADITION technologies AG
Schwarzwaldstraße 78b
79117 Freiburg
http://www.adition.com
T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
SUPPORT +49 / (0)1805 - ADITION
(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
UStIDNr.: DE 218 858 434
Well, I did that and rebuild all pig jars with -Dhadoopversion=23 but am
still getting errors - this time around it's errors with opening an
iterator on alias b again, sigh...
I have singled out one of the similarly failing tests and pasted the output:
http://pastebin.com/reKjcX0p
The test case failing looks like this:
http://pastebin.com/L5seh0ka
And INPUT_FILE gets created correctly through @BeforeClass
http://pastebin.com/wKrH7i50
If I run the script in the grunt shell invoked like this
pig -x local -propertyFile
/home/schwenk/dev/hadoop/trunk/pig/conf/adition.properties
everything works fine and I can execute the script
grunt> a = LOAD 'input.dat' AS (id:int, grp:int, additional:int,
referer:chararray);
b = FILTER a BY com.adition.pig.filtering.string.CONTAINS(referer, 'obama');
grunt> dump b;
...
2012-07-03 18:11:23,032 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1
(4,323,242,http://www.google.com/url&url=http%3A%2F%2Fwww.tagesschau.de&q=barack%20obama)
(5,423,342,http://www.google.com/url&url=http%3A%2F%2Fwww.bild.de&q=obama)
and verify that the output is correct.
Any further advice on this?
Thanks,
Johannes
Am 27.06.2012 22:24, schrieb Daniel Dai:
> Do "ant clean".
>
> On Thu, Jun 21, 2012 at 1:16 AM, Johannes Schwenk <
> [EMAIL PROTECTED]> wrote:
>
>> Am 20.06.2012 20:21, schrieb Daniel Dai:
>>> Have you recompiled Pig using -Dhadoopversion=23?
>>
>> No, thanks that solved the issue at hand!
>>
>> I have other errors though. My test cases return errors like
>>
>> http://pastebin.com/XpszYeu2
>>
>> So I am still doing something wrong it seems?!
>>
>> Thanks,
>> Johannes
>>
>>
>>> On Wed, Jun 20, 2012 at 4:38 AM, Johannes Schwenk <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Hi list,
>>>>
>>>> I have written two UDFs that run fine with Hadoop 0.20.2 and Pig 0.10.0.
>>>> I am trying to switch from Hadoop 0.20.2 to 0.23 and am encountering
>>>> some errors on running my tests:
>>>>
>>>> http://pastebin.com/JwHRQ9RQ
>>>>
>>>> Might this be a simple error in configuration?
>>>>
>>>> Johannes Schwenk
>>>>
>>>> --
>>>> Softwareentwickler (Reporting)
>>>> ________________________________________________________
>>>>
>>>> ADITION technologies AG
>>>> Schwarzwaldstraße 78b
>>>> 79117 Freiburg
>>>>
>>>> http://www.adition.com
>>>>
>>>> T +49 / (0)761 / 88147 - 30
>>>> F +49 / (0)761 / 88147 - 77
>>>> SUPPORT +49 / (0)1805 - ADITION
>>>>
>>>> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>>>>
>>>> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
>>>> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus
>> Schlüter
>>>> Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
>>>> UStIDNr.: DE 218 858 434
>>>>
>>>>
>>>
>>
>>
>>
>> Johannes Schwenk
>>
>> --
>> Softwareentwickler (Reporting)
>> ________________________________________________________
>>
>> ADITION technologies AG
>> Schwarzwaldstraße 78b
>> 79117 Freiburg
>>
>> http://www.adition.com
>>
>> T +49 / (0)761 / 88147 - 30
>> F +49 / (0)761 / 88147 - 77
>> SUPPORT +49 / (0)1805 - ADITION
>>
>> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>>
>> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
>> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
>> Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
>> UStIDNr.: DE 218 858 434
>>
>>
>
Johannes Schwenk
--
Softwareentwickler (Reporting)
________________________________________________________
ADITION technologies AG
Schwarzwaldstraße 78b
79117 Freiburg
http://www.adition.com
T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
SUPPORT +49 / (0)1805 - ADITION
(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
UStIDNr.: DE 218 858 434
