Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> jsonStorage and pig maps, not sure whats wrong with this?


+
Robert McCarthy 2013-02-21, 10:55
+
Dmitriy Ryaboy 2013-02-27, 09:26
+
Robert McCarthy 2013-03-01, 15:30
Copy link to this message
-
Re: jsonStorage and pig maps, not sure whats wrong with this?
Ah, pig 11 stores something but this seems to be generating an extra
"map_0" entry not the name value pair I was expecting.

I am getting

{"timestamp":"2013-01-31","map_0":{"@tags":"web"}}
{"timestamp":"2013-02-05","map_0":{"@tags":"web"}}

what I was hoping for was:

{"timestamp":"2013-01-31","@tags":"web"}
{"timestamp":"2013-02-05","@tags":"web"}

Anybody know how I can achieve this? I can't use

GENERATE platform as @tags
or
GENERATE platform as "@tags" or GENERATE platform as '@tags'

as it complains about the " ' @. Thought the TOMAP('@tags',platform) was a
clever work around.

On 1 March 2013 15:30, Robert McCarthy <[EMAIL PROTECTED]>wrote:

> Hi Dmitriy,
>
> This looks like it was a bug in pig 10 within the JsonStorage class as the
> same code now works in pig 11.
>
> The script is is
> A = load '/user/rob/events' USING PigStorage('\t') AS
> (dt_dt:chararray,platform:chararray);
> X = FOREACH A GENERATE dt_dt as timestamp,TOMAP('@tags',platform);
> STORE X INTO '/user/rob/events.json' USING JsonStorage;
>
> the data can be anything that has two columns that conform to chararray,
> it doesn't matter.
>
>
> On 27 February 2013 09:26, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:
>
>> Sounds odd. Can you send a complete script that reproduces the error
>> (include sample data and load statements).
>>
>>
>> On Thu, Feb 21, 2013 at 2:55 AM, Robert McCarthy <
>> [EMAIL PROTECTED]> wrote:
>>
>> > If I have some information in A, that contains dt_dt and platform, I
>> want
>> > to store it in a different json format,
>> > So I can create a simple new bag like this
>> > X = FOREACH A GENERATE dt_dt as timestamp;
>> > and it stores ok in JsonStorage
>> > STORE X INTO '/user/rob/events.json' USING JsonStorage;
>> >
>> > I want to create a json object containing a pig map, because in the json
>> > object I want "@tags": platform,
>> > I can create the map ok like this..
>> > X = FOREACH A GENERATE dt_dt as timestamp,TOMAP('@tags',platform);
>> >
>> > Describe and illustrate all show the pig data as expected.
>> >
>> > describe x
>> > x: {timestamp: chararray,map[]}
>> >
>> > illustrate x
>> > --------------------------------------------------
>> > | x     | timestamp:chararray      | :map        |
>> > --------------------------------------------------
>> > |       | 2013-02-05               | {@tags=web} |
>> > --------------------------------------------------
>> >
>> > but it will not store using JsonStorage
>> > STORE X INTO '/user/rob/events.json' USING JsonStorage;
>> >
>> > Failed to parse: <line 1, column 15>  mismatched input '[' expecting
>> > IDENTIFIER_L
>> >
>> > I have tried without the @ to see if that was causing the error but get
>> the
>> > same result.
>> > Any help much appreciated.
>> >
>> > Rob,
>> >
>>
>
>