Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop >> mail # user >> Creating compressed data with scooop


+
Santosh Achhra 2013-01-07, 11:34
+
Jarek Jarcec Cecho 2013-01-07, 12:31
+
Santosh Achhra 2013-01-08, 05:21
+
Jarek Jarcec Cecho 2013-01-08, 07:36
+
Santosh Achhra 2013-01-08, 08:01
+
Jarek Jarcec Cecho 2013-01-08, 11:09
+
Santosh Achhra 2013-01-08, 14:33
+
Jarek Jarcec Cecho 2013-01-08, 15:43
+
Santosh Achhra 2013-01-08, 16:09
+
Jarek Jarcec Cecho 2013-01-09, 10:01
+
Santosh Achhra 2013-01-09, 11:45
+
Jarek Jarcec Cecho 2013-01-10, 09:40
Copy link to this message
-
Re: Creating compressed data with scooop
Hi Jarcec,

Are you referring to task logs ? in below screen. When I click on "All" or
"Last 4KB" or "Last 8KB', the page times out :-( Not sure why

task_201301072354_0114_m_000000 attempts for
0114_1357651667175_saachhra<http://sl73caeq02.visa.com:50030/jobdetailshistory.jsp?logFile=file:/hadoop1/logs/hadoop/history/done/sl73caeq02.visa.com_1357602885590_/2013/00/08/000000/job_201301072354_0114_1357651667175_saachhra_optqc5d.TQC_ELIG_TRAN_HIST.jar>
Task
IdStart TimeFinish Time HostErrorTask Logs Counters
attempt_201301072354_0114_m_000000_0 8/01 13:27:538/01 13:27:58 (5sec)
/default/host Last
4KB<http://sl73caeq03.visa.com:50060/tasklog?attemptid=attempt_201301072354_0114_m_000000_0&start=-4097>
Last 8KB<http://sl73caeq03.visa.com:50060/tasklog?attemptid=attempt_201301072354_0114_m_000000_0&start=-8193>
All<http://sl73caeq03.visa.com:50060/tasklog?attemptid=attempt_201301072354_0114_m_000000_0&all=true>
18<http://sl73caeq02.visa.com:50030/taskstatshistory.jsp?attemptid=attempt_201301072354_0114_m_000000_0&logFile=file:/hadoop1/logs/hadoop/history/done/sl73caeq02.visa.com_1357602885590_/2013/00/08/000000/job_201301072354_0114_1357651667175_saachhra_optqc5d.TQC_ELIG_TRAN_HIST.jar>
However I got the location on linux here it was trying to look.  I am
afraid I wont be able to attach file or put full contents in this email :-(
Is this what you were looking for (See below) ?
- <property>
  <name>mapred.compress.map.output</name>
  <value>true</value>
  <source>mapred-site.xml</source>
  <source>/data/mapred/jt/jobTracker/job_201301072354_0114.xml</source>
  </property>

- <property>
  <name>io.seqfile.lazydecompress</name>
  <value>true</value>
  <source>core-default.xml</source>
  <source>/data/mapred/jt/jobTracker/job_201301072354_0114.xml</source>
  </property>

 - <property>
  <name>io.compression.codecs</name>

<value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
  <source>core-site.xml</source>
  <source>/data/mapred/jt/jobTracker/job_201301072354_0114.xml</source>
  </property>
- <property>
  <name>mapred.output.compression.type</name>
  <value>BLOCK</value>
  <source>mapred-site.xml</source>
  <source>/data/mapred/jt/jobTracker/job_201301072354_0114.xml</source>
  </property>
- <property>
  <name>mapred.output.compression.codec</name>
  <value>org.apache.hadoop.io.compress.GzipCodec</value>
  <source>programatically</source>
  <source>/data/mapred/jt/jobTracker/job_201301072354_0114.xml</source>
  </property>

- <property>
  <name>mapred.map.output.compression.codec</name>
  <value>org.apache.hadoop.io.compress.SnappyCodec</value>
  <source>mapred-site.xml</source>
  <source>/data/mapred/jt/jobTracker/job_201301072354_0114.xml</source>
  </property>
- <property>
  <name>io.seqfile.compress.blocksize</name>
  <value>1000000</value>
  <source>core-default.xml</source>
  <source>/data/mapred/jt/jobTracker/job_201301072354_0114.xml</source>
  </property>

- <property>
  <name>dfs.image.compress</name>
  <value>false</value>
  <source>hdfs-default.xml</source>
  <source>/data/mapred/jt/jobTracker/job_201301072354_0114.xml</source>
  </property>

- <property>
  <name>dfs.image.compression.codec</name>
  <value>org.apache.hadoop.io.compress.DefaultCodec</value>
  <source>hdfs-default.xml</source>
  <source>/data/mapred/jt/jobTracker/job_201301072354_0114.xml</source>
  </property>

Good wishes,always !
Santosh
On Thu, Jan 10, 2013 at 5:40 PM, Jarek Jarcec Cecho <[EMAIL PROTECTED]>wrote:

> Hi Santosh,
> almost :-). When you click on this page on the "map" link you should get
> page with map tasks. When you click on one arbitrary task you should get
> table with all corresponding task attempts. On this page you should see
> link to log.
>
> Jarcec
>
> On Wed, Jan 09, 2013 at 07:45:44PM +0800, Santosh Achhra wrote:
> > Hi Jarcec,
> >
> > Hopefully this is log which you had requested.