Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> has bzip2 compression been deprecated?


Copy link to this message
-
Re: has bzip2 compression been deprecated?
Hi Tony
       Adding on to Harsh's comments. If you want the generated sequence
files to be utilized by a hive table. Define your hive table as

CREATE EXTERNAL TABLE tableNAme(col1 INT, c0l2 STRING)
...
...
....
STORED AS SEQUENCEFILE;
Regards
Bejoy.K.S

On Mon, Jan 9, 2012 at 10:32 PM, alo.alt <[EMAIL PROTECTED]> wrote:

> Tony,
>
> snappy is also available:
> http://code.google.com/p/hadoop-snappy/
>
> best,
>  Alex
>
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
>
> On Jan 9, 2012, at 8:49 AM, Harsh J wrote:
>
> > Tony,
> >
> > * Yeah, SequenceFiles aren't human-readable, but "fs -text" can read it
> out (instead of a plain "fs -cat"). But if you are gonna export your files
> into a system you do not have much control over, probably best to have the
> resultant files not be in SequenceFile/Avro-DataFile format.
> > * Intermediate (M-to-R) files use a custom IFile format these days,
> which is built purely for that purpose.
> > * Hive can use SequenceFiles very well. There is also documented info on
> this in the Hive's wiki pages (Check the DDL pages, IIRC).
> >
> > On 09-Jan-2012, at 9:44 PM, Tony Burton wrote:
> >
> >> Thanks for the quick reply and the clarification about the
> documentation.
> >>
> >> Regarding sequence files: am I right in thinking that they're a good
> choice for intermediate steps in chained MR jobs, or for file transfer
> between the Map and the Reduce phases of a job; but they shouldn't be used
> for human-readable files at the end of one or more MapReduce jobs? How
> about if the only use a job's output is analysis via Hive - can Hive create
> tables from sequence files?
> >>
> >> Tony
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Harsh J [mailto:[EMAIL PROTECTED]]
> >> Sent: 09 January 2012 15:34
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: has bzip2 compression been deprecated?
> >>
> >> Bzip2 is pretty slow. You probably do not want to use it, even if it
> does file splits (a feature not available in the stable line of 0.20.x/1.x,
> but available in 0.22+).
> >>
> >> To answer your question though, bzip2 was removed from that document
> cause it isn't a native library (its pure Java). I think bzip2 was added
> earlier due to an oversight, as even 0.20 did not have a native bzip2
> library. This change in docs does not mean that BZip2 is deprecated -- it
> is still fully supported and available in the trunk as well. See
> https://issues.apache.org/jira/browse/HADOOP-6292 for the doc update
> changes that led to this.
> >>
> >> The best way would be to use either:
> >>
> >> (a) Hadoop sequence files with any compression codec of choice (best
> would be lzo, gz, maybe even snappy). This file format is built for HDFS
> and MR and is splittable. Another choice would be Avro DataFiles from the
> Apache Avro project.
> >> (b) LZO codecs for Hadoop, via https://github.com/toddlipcon/hadoop-lzo(and hadoop-lzo-packager for packages). This requires you to run indexing
> operations before the .lzo can be made splittable, but works great with
> this extra step added.
> >>
> >> On 09-Jan-2012, at 7:17 PM, Tony Burton wrote:
> >>
> >>> Hi,
> >>>
> >>> I'm trying to work out which compression algorithm I should be using
> in my MapReduce jobs.  It seems to me that the best solution is a
> compromise between speed, efficiency and splittability. The only
> compression algorithm to handle file splits (according to Hadoop: The
> Definitive Guide 2nd edition p78 etc) is bzip2, at the expense of
> compression speed.
> >>>
> >>> However, I see from the documentation at
> http://hadoop.apache.org/common/docs/current/native_libraries.html that
> the bzip2 library is no longer mentioned, and hasn't been since version
> 0.20.0, see
> http://hadoop.apache.org/common/docs/r0.20.0/native_libraries.html -
> however the bzip2 Codec is still in the API at
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/compress/BZip2Codec.html
> .
> >>>
> >>> Has bzip2 support been removed from Hadoop, or will it be removed soon?