Good point sir. If Pig fits into Panshul's requirements then it's a much
better option.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 7, 2013 at 5:25 PM, Damien Hardy <[EMAIL PROTECTED]> wrote:

> Hello,
> Why not using a PIG script for that ?
> make the json file available on HDFS
> Load with
>
> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/builtin/JsonLoader.html
> Store with
>
> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/backend/hadoop/hbase/HBaseStorage.html
>
> http://pig.apache.org/docs/r0.10.0/
>
> Cheers,
>
> --
> Damien
>

Hello,

Thank you for the replies.

I have not used pig yet. I am looking into it. I wanted to implement both
the approaches.
Are pig scripts maintainable? Because the Json structure that I will be
receiving will be changing quite often. Almost 3 times a month.
I will be processing 24 million Json files per month.
I am getting one big file with almost 3 million Json files aggregated. One
Json per line. I need to process this file and store all values into HBase.

Thanking You,
On Thu, Feb 7, 2013 at 12:59 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:

> Good point sir. If Pig fits into Panshul's requirements then it's a much
> better option.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 7, 2013 at 5:25 PM, Damien Hardy <[EMAIL PROTECTED]>
> wrote:
>
> > Hello,
> > Why not using a PIG script for that ?
> > make the json file available on HDFS
> > Load with
> >
> >
> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/builtin/JsonLoader.html
> > Store with
> >
> >
> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/backend/hadoop/hbase/HBaseStorage.html
> >
> > http://pig.apache.org/docs/r0.10.0/
> >
> > Cheers,
> >
> > --
> > Damien
> >
>

--
Regards,
Ouch Whisper
010101010101

Size is not a prob, frequently changing schema might be.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 7, 2013 at 6:25 PM, Panshul Whisper <[EMAIL PROTECTED]>wrote:

> Hello,
>
> Thank you for the replies.
>
> I have not used pig yet. I am looking into it. I wanted to implement both
> the approaches.
> Are pig scripts maintainable? Because the Json structure that I will be
> receiving will be changing quite often. Almost 3 times a month.
> I will be processing 24 million Json files per month.
> I am getting one big file with almost 3 million Json files aggregated. One
> Json per line. I need to process this file and store all values into HBase.
>
> Thanking You,
>
>
>
>
> On Thu, Feb 7, 2013 at 12:59 PM, Mohammad Tariq <[EMAIL PROTECTED]>
> wrote:
>
> > Good point sir. If Pig fits into Panshul's requirements then it's a much
> > better option.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Thu, Feb 7, 2013 at 5:25 PM, Damien Hardy <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hello,
> > > Why not using a PIG script for that ?
> > > make the json file available on HDFS
> > > Load with
> > >
> > >
> >
> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/builtin/JsonLoader.html
> > > Store with
> > >
> > >
> >
> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/backend/hadoop/hbase/HBaseStorage.html
> > >
> > > http://pig.apache.org/docs/r0.10.0/
> > >
> > > Cheers,
> > >
> > > --
> > > Damien
> > >
> >
>
>
>
> --
> Regards,
> Ouch Whisper
> 010101010101
>

Pig is like any other scripting language. Therefore maintainable.

Your script defines the schema of the data set and then does the processing you tell it to do.

On Feb 7, 2013, at 6:55 AM, Panshul Whisper <[EMAIL PROTECTED]> wrote:

> Hello,
>
> Thank you for the replies.
>
> I have not used pig yet. I am looking into it. I wanted to implement both
> the approaches.
> Are pig scripts maintainable? Because the Json structure that I will be
> receiving will be changing quite often. Almost 3 times a month.
> I will be processing 24 million Json files per month.
> I am getting one big file with almost 3 million Json files aggregated. One
> Json per line. I need to process this file and store all values into HBase.
>
> Thanking You,
>
>
>
>
> On Thu, Feb 7, 2013 at 12:59 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>
>> Good point sir. If Pig fits into Panshul's requirements then it's a much
>> better option.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Thu, Feb 7, 2013 at 5:25 PM, Damien Hardy <[EMAIL PROTECTED]>
>> wrote:
>>
>>> Hello,
>>> Why not using a PIG script for that ?
>>> make the json file available on HDFS
>>> Load with
>>>
>>>
>> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/builtin/JsonLoader.html
>>> Store with
>>>
>>>
>> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/backend/hadoop/hbase/HBaseStorage.html
>>>
>>> http://pig.apache.org/docs/r0.10.0/
>>>
>>> Cheers,
>>>
>>> --
>>> Damien
>>>
>>
>
>
>
> --
> Regards,
> Ouch Whisper
> 010101010101

The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com