Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> LZO & Pig (Elephantbird?)


Copy link to this message
-
Re: LZO & Pig (Elephantbird?)
Alright, thanks! I'm going to give that a try and will let you know how it goes.

Cheers, Evert

----- Reply message -----
From: "Dmitriy Ryaboy" <[EMAIL PROTECTED]>
Date: Thu, Jan 13, 2011 8:27 pm
Subject: LZO & Pig (Elephantbird?)
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>

Depends on your definition of ready. I haven't put it into production and
there's a bit of clean-up left, but as far as I know there are no critical
bugs; just some interface issues which are shared between the current master
and the version for Pig 0.8.

Pig 0.8 should work with Hadoop 0.20.1; I've been working with it under CDH2
(which is a patched 0.20.1).

-Dmitriy

On Thu, Jan 13, 2011 at 7:43 AM, Evert Lammerts <[EMAIL PROTECTED]>wrote:

> > Are you trying to load protobufs or thrift files, or do you just want
> > Lzo
> > support?
>
> Protobufs would be nice, but Elephant Bird is not ready yet for Pig 0.7 /
> 0.8, right?
>
> > If you just want plain text lzo loading, the loaders in the
> > pig-08
> > branch totally work.
>
> Does Pig 0.8 work with Hadoop 0.20.1 as well?
>
> Thanks for the support!
> Evert
>
> >
> > Let me know if you have any issues.
> >
> > D
> >
> > On Wed, Jan 12, 2011 at 12:23 PM, Tyler Coffin <[EMAIL PROTECTED]> wrote:
> >
> > > There's a fork of elephant-bird where pig-8 support is being worked
> > on:
> > > https://github.com/dvryaboy/elephant-bird/tree/pig-08
> > >
> > > I haven't given it a shot yet.
> > >
> > > -----Original Message-----
> > > From: Evert Lammerts [mailto:[EMAIL PROTECTED]]
> > > Sent: January 12, 2011 15:10
> > > To: '[EMAIL PROTECTED]'
> > > Subject: LZO & Pig (Elephantbird?)
> > >
> > > Hello list,
> > >
> > > I've installed the LZO codecs (https://github.com/kevinweil/hadoop-
> > lzo)
> > > and
> > > now I'm looking into using LZO in Pig. Elephant Bird
> > > (https://github.com/kevinweil/elephant-bird) seems to provide some
> > nice
> > > prefab loaders, but it's requirements do not fit out Hadoop
> > installation
> > > (we're on CDH3b2 with Pig 0.7, EB cannot be used with anything >
> > 0.6). Also
> > > the need for Thrift 0.2 is unclear to me - Thrift is now at 0.5.
> > >
> > > Now I did find this project, http://code.google.com/p/hadoop-gpl-
> > packing/,
> > > saying EB can handle even Pig 0.8. This confuses me - can I or can I
> > not
> > > use
> > > Elephant Bird with Pig 0.7, or even upgrade to Pig 0.8?
> > >
> > > Since EB is probably not an option, does anybody have some pointers
> > on how
> > > to use LZO'ed files with Pig?
> > >
> > > Thanks!
> > >
> > > Evert Lammerts
> > >
> > > ---------------------------------------------------------------------
> > > This transmission (including any attachments) may contain
> > confidential
> > > information, privileged material (including material protected by the
> > > solicitor-client or other applicable privileges), or constitute non-
> > public
> > > information. Any use of this information by anyone other than the
> > intended
> > > recipient is prohibited. If you have received this transmission in
> > error,
> > > please immediately reply to the sender and delete this information
> > from your
> > > system. Use, dissemination, distribution, or reproduction of this
> > > transmission by unintended recipients is not authorized and may be
> > unlawful.
> > >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB