Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - oddness with large numeric map[] values


Copy link to this message
-
RE: oddness with large numeric map[] values
Santhosh Srinivasan 2009-07-19, 06:23
I just read your email all over again. The reason for the failure is the
following.

1. [apache#2000000000000000000000zzz,foo#foo1] - Here the value for the
key apache is treated as a string.

2. [apache#2000000000000000000000,foo#foo1] - Here the value for the key
apache is treated as an integer. Since 2000000000000000000000 is too big
to fit into an integer it failed and inserted a null. Try adding an L at
the end of 2000000000000000000000, i.e., 2000000000000000000000L

Santhosh

-----Original Message-----
From: Guy Bayes [mailto:[EMAIL PROTECTED]]
Sent: Saturday, July 18, 2009 11:21 PM
To: [EMAIL PROTECTED]
Subject: Re: oddness with large numeric map[] values

this is reproducible (at least for me) with any large number in the
value
column of any map schema declaration. Tried it on pig v0.2 and 0.3

On Sat, Jul 18, 2009 at 11:09 PM, Santhosh Srinivasan
<[EMAIL PROTECTED]>wrote:

> In the second case, there is a warning message:
>
> - Encountered Warning FIELD_DISCARDED_TYPE_CONVERSION_FAILED 1
time(s).
>
> The conversion of the text data into a map failed for some reason. If
> you examine the system logs on the job tracker UI, you should probably
> see why the conversion failed. You can also achieve the same result by
> turning warning aggregation off with the -w option.
>
> Santhosh
>
>
> -----Original Message-----
> From: Guy Bayes [mailto:[EMAIL PROTECTED]]
> Sent: Saturday, July 18, 2009 9:37 PM
> To: [EMAIL PROTECTED]
> Subject: oddness with large numeric map[] values
>
> Hello all, new to this list, new to pig, running into some odd
behavior
> with
> map[] data types.
>
> Please forgive me if these are known issues or problems with my
syntax,
>
> What am i doing wrong here? missing some cast somewhere?
>
> This works:
>
> grunt> cat data2
> [apache#2000000000000000000000zzz,foo#foo1]
> grunt> T1 = load 'data2' as (f1:map[]);
> grunt> dump T1;
> 2009-07-18 21:35:11,871 [Thread-11] WARN
> org.apache.hadoop.mapred.JobClient
> - Use GenericOptionsParser for parsing the arguments. Applications
> should
> implement Tool for the same.
> 2009-07-18 21:35:21,889 [main] INFO
>
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
> uncher
> - 100% complete
> 2009-07-18 21:35:21,889 [main] INFO
>
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
> uncher
> - Success!
> ([foo#foo1,apache#2000000000000000000000zzz])
>
> This doesn't return anything.
>
> grunt> cat data
> [apache#2000000000000000000000,foo#foo1]
> grunt> T1 = load 'data' as (f1:map[]);
> grunt> dump T1;
> 2009-07-18 21:36:09,873 [Thread-21] WARN
> org.apache.hadoop.mapred.JobClient
> - Use GenericOptionsParser for parsing the arguments. Applications
> should
> implement Tool for the same.
> 2009-07-18 21:36:19,886 [main] WARN
>
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
> uncher
> - Encountered Warning FIELD_DISCARDED_TYPE_CONVERSION_FAILED 1
time(s).
> 2009-07-18 21:36:19,886 [main] INFO
>
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
> uncher
> - 100% complete
> 2009-07-18 21:36:19,886 [main] INFO
>
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLa
> uncher
> - Success!
> ()
>

--
you may be acquainted with the night
but i have seen the darkness in the day
and you must know it is a terrifying sight...