Hi,
  Im  looking for  a solution to load/store   lzo compressed files   for Pig-0.9.1 .   Any pointers would be useful.  

Thanks,

Sam William
[EMAIL PROTECTED]

Hi Sam,

Elephantbird might be something you are looking for:
https://github.com/kevinweil/elephant-bird

Thanks,
Prashant

On Mon, Jan 23, 2012 at 1:42 PM, Sam William <[EMAIL PROTECTED]> wrote:

> Hi,
>  Im  looking for  a solution to load/store   lzo compressed files   for
> Pig-0.9.1 .   Any pointers would be useful.
>
> Thanks,
>
> Sam William
> [EMAIL PROTECTED]
>
>
>
>

I did read about this.  Seems like it was experimental even for 0.8 .   I wasnt sure of 0.9.    Moreover , I couldn't build it  without protocol buffer support.   The noproto target seems to have been removed.

Sam
On Jan 23, 2012, at 1:49 PM, Prashant Kommireddi wrote:

> Hi Sam,
>
> Elephantbird might be something you are looking for:
> https://github.com/kevinweil/elephant-bird
>
> Thanks,
> Prashant
>
> On Mon, Jan 23, 2012 at 1:42 PM, Sam William <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>> Im  looking for  a solution to load/store   lzo compressed files   for
>> Pig-0.9.1 .   Any pointers would be useful.
>>
>> Thanks,
>>
>> Sam William
>> [EMAIL PROTECTED]
>>
>>
>>
>>

Sam William
[EMAIL PROTECTED]

btw, for simple use cases, you can load/store lzo files using PigStorage().
While stor

Only disadvantage is that files are not splittable.

grunt> set output.compression.enabled true;
grunt> set output.compression.codec com.hadoop.compression.lzo.LzopCodec;
grunt> x = load 'x.txt' using PigStorage();
grunt> store x into 'xout';
grunt> fs -ls xout;
Found 1 items
-rw-r--r--   3 raghu users         68 2012-01-24 00:29
/user/raghu/xout/part-m-00000.lzo
Raghu.

On Mon, Jan 23, 2012 at 1:42 PM, Sam William <[EMAIL PROTECTED]> wrote:

> Hi,
>  Im  looking for  a solution to load/store   lzo compressed files   for
> Pig-0.9.1 .   Any pointers would be useful.
>
> Thanks,
>
> Sam William
> [EMAIL PROTECTED]
>
>
>
>

It's far from experimental, Twitter's been running EB on pig 0.8 and 0.9
for a year now.

But you do need the protocol buffers 2.3 installed to build. We might just
check in the generated code for the required bits so you don't have to some
time soon..

Dmitriy
On Mon, Jan 23, 2012 at 2:00 PM, Sam William <[EMAIL PROTECTED]> wrote:

> I did read about this.  Seems like it was experimental even for 0.8 .   I
> wasnt sure of 0.9.    Moreover , I couldn't build it  without protocol
> buffer support.   The noproto target seems to have been removed.
>
> Sam
> On Jan 23, 2012, at 1:49 PM, Prashant Kommireddi wrote:
>
> > Hi Sam,
> >
> > Elephantbird might be something you are looking for:
> > https://github.com/kevinweil/elephant-bird
> >
> > Thanks,
> > Prashant
> >
> > On Mon, Jan 23, 2012 at 1:42 PM, Sam William <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi,
> >> Im  looking for  a solution to load/store   lzo compressed files   for
> >> Pig-0.9.1 .   Any pointers would be useful.
> >>
> >> Thanks,
> >>
> >> Sam William
> >> [EMAIL PROTECTED]
> >>
> >>
> >>
> >>
>
> Sam William
> [EMAIL PROTECTED]
>
>
>
>

Dmitriy/ Raghu,
  Thanks very much.

Sam
On Jan 23, 2012, at 10:29 PM, Dmitriy Ryaboy wrote:

> It's far from experimental, Twitter's been running EB on pig 0.8 and 0.9
> for a year now.
>
> But you do need the protocol buffers 2.3 installed to build. We might just
> check in the generated code for the required bits so you don't have to some
> time soon..
>
> Dmitriy
>
>
> On Mon, Jan 23, 2012 at 2:00 PM, Sam William <[EMAIL PROTECTED]> wrote:
>
>> I did read about this.  Seems like it was experimental even for 0.8 .   I
>> wasnt sure of 0.9.    Moreover , I couldn't build it  without protocol
>> buffer support.   The noproto target seems to have been removed.
>>
>> Sam
>> On Jan 23, 2012, at 1:49 PM, Prashant Kommireddi wrote:
>>
>>> Hi Sam,
>>>
>>> Elephantbird might be something you are looking for:
>>> https://github.com/kevinweil/elephant-bird
>>>
>>> Thanks,
>>> Prashant
>>>
>>> On Mon, Jan 23, 2012 at 1:42 PM, Sam William <[EMAIL PROTECTED]>
>> wrote:
>>>
>>>> Hi,
>>>> Im  looking for  a solution to load/store   lzo compressed files   for
>>>> Pig-0.9.1 .   Any pointers would be useful.
>>>>
>>>> Thanks,
>>>>
>>>> Sam William
>>>> [EMAIL PROTECTED]
>>>>
>>>>
>>>>
>>>>
>>
>> Sam William
>> [EMAIL PROTECTED]
>>
>>
>>
>>

Sam William
[EMAIL PROTECTED]