Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # user - Creating compressed data with scooop


+
Santosh Achhra 2013-01-07, 11:34
+
Jarek Jarcec Cecho 2013-01-07, 12:31
+
Santosh Achhra 2013-01-08, 05:21
+
Jarek Jarcec Cecho 2013-01-08, 07:36
+
Santosh Achhra 2013-01-08, 08:01
Copy link to this message
-
Re: Creating compressed data with scooop
Jarek Jarcec Cecho 2013-01-08, 11:09
I do not see any discrepancy here nor in the TaskTracker configuration. The mapreduce output compression should be working. Property:

* io.compression.codecs - Contains codes you were using
* mapred.output.compress - is set to true and is not set to final in TT config
* mapred.output.compression.codec - seems to be set correctly

Are you sure that your TaskTrackers were started with this configuration?

Would you mind sharing the map task output? Just to be sure that there  won't be anything suspicious.

Jarcec

On Tue, Jan 08, 2013 at 04:01:35PM +0800, Santosh Achhra wrote:
> Thank you Jarcec,
>
> I am posting values for few rows, Are you looking for any specific row ?
> Thanks once again for your help.
>
> mapred.compress.map.output true
> io.seqfile.lazydecompress true
> io.compression.codecs
> org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec
> mapred.output.compression.type BLOCK
> mapred.output.compression.codec org.apache.hadoop.io.compress.SnappyCodec
> io.seqfile.compress.blocksize 1000000
> dfs.image.compress false
> mapred.output.compress true
> dfs.image.compression.codec org.apache.hadoop.io.compress.DefaultCodec
>
> Good wishes,always !
> Santosh
>
>
> On Tue, Jan 8, 2013 at 3:36 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:
>
> > L from JobTracker web ui by clicking at you job details (column
+
Santosh Achhra 2013-01-08, 14:33
+
Jarek Jarcec Cecho 2013-01-08, 15:43
+
Santosh Achhra 2013-01-08, 16:09
+
Jarek Jarcec Cecho 2013-01-09, 10:01
+
Santosh Achhra 2013-01-09, 11:45
+
Jarek Jarcec Cecho 2013-01-10, 09:40
+
Santosh Achhra 2013-01-10, 12:47