Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - oddness with large numeric map[] values
Guy Bayes
2009-07-19, 04:36
Santhosh Srinivasan
2009-07-19, 06:09
Guy Bayes
2009-07-19, 06:21
Santhosh Srinivasan
2009-07-19, 06:23
Guy Bayes
2009-07-19, 06:49
Santhosh Srinivasan
2009-07-19, 15:50
Guy Bayes
2009-07-19, 16:28
Fatal.error
2009-07-19, 19:15
zjffdu
2009-07-19, 23:39
Copy link to this message
-
RE: oddness with large numeric map[] values
Santhosh Srinivasan 2009-07-19, 19:19
Its already captured as part of Pig-724(https://issues.apache.org/jira/browse/PIG-724) which in turn is linked to Pig-880.

Thanks,
Santhosh

-----Original Message-----
From: Fatal.error [mailto:[EMAIL PROTECTED]]
Sent: Sunday, July 19, 2009 12:16 PM
To: [EMAIL PROTECTED]
Subject: Re: oddness with large numeric map[] values

So should I leave this alone or file a seperate bug report you think?
Thanks
Guy

On Jul 19, 2009, at 8:50 AM, "Santhosh Srinivasan" <[EMAIL PROTECTED]>  
wrote:

> With Pig-880 (https://issues.apache.org/jira/browse/PIG-880), the  
> value per key in the text data will be treated as bytearray. When  
> Pig-880 is committed (implicit/explicit) casts will be required to  
> interpret the data.
>
> Thanks,
> Santhosh
>
> -----Original Message-----
> From: zjffdu [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, July 19, 2009 4:40 PM
> To: [EMAIL PROTECTED]
> Subject: RE: oddness with large numeric map[] values
>
> +1， I am also curious to know about this.  I think it make no sense  
> if
> there's no such feature.
>
>
>
> -----Original Message-----
> From: Guy Bayes [mailto:[EMAIL PROTECTED]]
> Sent: 2009年7月18日 23:50
> To: [EMAIL PROTECTED]
> Subject: Re: oddness with large numeric map[] values
>
> yup the problem is that the actual data actually contains strings  
> that look
> like numbers occasionally.
>
> is there any way to tell apache to treat it like a string always?
>
> On Sat, Jul 18, 2009 at 11:23 PM, Santhosh Srinivasan
> <[EMAIL PROTECTED]>wrote:
>
>> I just read your email all over again. The reason for the failure  
>> is the
>> following.
>>
>> 1. [apache#2000000000000000000000zzz,foo#foo1] - Here the value for  
>> the
>> key apache is treated as a string.
>>
>> 2. [apache#2000000000000000000000,foo#foo1] - Here the value for  
>> the key
>> apache is treated as an integer. Since 2000000000000000000000 is  
>> too big
>> to fit into an integer it failed and inserted a null. Try adding an  
>> L at
>> the end of 2000000000000000000000, i.e., 2000000000000000000000L
>>
>> Santhosh
>>
>> -----Original Message-----
>> From: Guy Bayes [mailto:[EMAIL PROTECTED]]
>> Sent: Saturday, July 18, 2009 11:21 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: oddness with large numeric map[] values
>>
>> this is reproducible (at least for me) with any large number in the
>> value
>> column of any map schema declaration. Tried it on pig v0.2 and 0.3
>>
>> On Sat, Jul 18, 2009 at 11:09 PM, Santhosh Srinivasan
>> <[EMAIL PROTECTED]>wrote:
>>
>>> In the second case, there is a warning message:
>>>
>>> - Encountered Warning FIELD_DISCARDED_TYPE_CONVERSION_FAILED 1
>> time(s).
>>>
>>> The conversion of the text data into a map failed for some reason.  
>>> If
>>> you examine the system logs on the job tracker UI, you should  
>>> probably
>>> see why the conversion failed. You can also achieve the same  
>>> result by
>>> turning warning aggregation off with the -w option.
>>>
>>> Santhosh
>>>
>>>
>>> -----Original Message-----
>>> From: Guy Bayes [mailto:[EMAIL PROTECTED]]
>>> Sent: Saturday, July 18, 2009 9:37 PM
>>> To: [EMAIL PROTECTED]
>>> Subject: oddness with large numeric map[] values
>>>
>>> Hello all, new to this list, new to pig, running into some odd
>> behavior
>>> with
>>> map[] data types.
>>>
>>> Please forgive me if these are known issues or problems with my
>> syntax,
>>>
>>> What am i doing wrong here? missing some cast somewhere?
>>>
>>> This works:
>>>
>>> grunt> cat data2
>>> [apache#2000000000000000000000zzz,foo#foo1]
>>> grunt> T1 = load 'data2' as (f1:map[]);
>>> grunt> dump T1;
>>> 2009-07-18 21:35:11,871 [Thread-11] WARN
>>> org.apache.hadoop.mapred.JobClient
>>> - Use GenericOptionsParser for parsing the arguments. Applications
>>> should
>>> implement Tool for the same.
>>> 2009-07-18 21:35:21,889 [main] INFO
>>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
 

>>> uncher
>>> - 100% complete
>>> 2009-07-18 21:35:21,889 [main] INFO