Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop, mail # dev - Review Request 13035: SQOOP-744: log4j configuration for generated mapreduce job


Copy link to this message
-
Re: Review Request 13035: SQOOP-744: log4j configuration for generated mapreduce job
Jarek Cecho 2013-08-19, 00:49


> On Aug. 14, 2013, 11:57 p.m., Jarek Cecho wrote:
> > execution/mapreduce/src/main/resources/META-INF/log4j.properties, lines 20-23
> > <https://reviews.apache.org/r/13035/diff/2/?file=330782#file330782line20>
> >
> >     I've tried the patch on a real cluster and got following output (please accept my apologies for the really long text):
> >    
> >     Task Logs: 'attempt_201308141631_0001_m_000001_0'
> >    
> >    
> >     stdout logs
> >    
> >    
> >     stderr logs
> >     2660 [OutputFormatLoader-consumer] INFO  org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor  - SqoopOutputFormatLoadExecutor consumer thread is starting
> >     2748 [OutputFormatLoader-consumer] INFO  org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor  - Running loader class org.apache.sqoop.job.etl.HdfsTextImportLoader
> >     2752 [main] INFO  org.apache.sqoop.job.mr.SqoopMapper  - Starting progress service
> >     2777 [main] INFO  org.apache.sqoop.job.mr.SqoopMapper  - Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor
> >     2782 [pool-2-thread-1] DEBUG org.apache.sqoop.job.mr.ProgressRunnable  - Auto-progress thread reporting progress
> >     3969 [main] INFO  org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor  - Using query: SELECT * FROM text WHERE 100001 <= id AND id < 200001
> >     32122 [main] INFO  org.apache.sqoop.job.mr.SqoopMapper  - Extractor has finished
> >     32129 [main] INFO  org.apache.sqoop.job.mr.SqoopMapper  - Stopping progress service
> >     32136 [main] INFO  org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor  - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
> >     34002 [OutputFormatLoader-consumer] INFO  org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor  - Loader has finished
> >     34002 [main] INFO  org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor  - SqoopOutputFormatLoadExecutor::SqoopRecordWriter is closed
> >    
> >    
> >     syslog logs
> >     2013-08-14 16:47:17,950 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
> >     2013-08-14 16:47:19,451 WARN org.apache.hadoop.conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
> >     2013-08-14 16:47:19,452 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId> >     2013-08-14 16:47:20,160 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
> >     2013-08-14 16:47:20,172 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@549b6220
> >     2013-08-14 16:47:20,585 INFO org.apache.hadoop.mapred.MapTask: Processing split: org.apache.sqoop.job.mr.SqoopSplit@23de4dd8
> >     2013-08-14 16:47:20,610 INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: SqoopOutputFormatLoadExecutor consumer thread is starting
> >     2013-08-14 16:47:20,698 INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: Running loader class org.apache.sqoop.job.etl.HdfsTextImportLoader
> >     2013-08-14 16:47:20,702 INFO org.apache.sqoop.job.mr.SqoopMapper: Starting progress service
> >     2013-08-14 16:47:20,727 INFO org.apache.sqoop.job.mr.SqoopMapper: Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor
> >     2013-08-14 16:47:20,732 DEBUG org.apache.sqoop.job.mr.ProgressRunnable: Auto-progress thread reporting progress
> >     2013-08-14 16:47:21,919 INFO org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor: Using query: SELECT * FROM text WHERE 100001 <= id AND id < 200001
> >     2013-08-14 16:47:50,072 INFO org.apache.sqoop.job.mr.SqoopMapper: Extractor has finished
> >     2013-08-14 16:47:50,079 INFO org.apache.sqoop.job.mr.SqoopMapper: Stopping progress service
> >     2013-08-14 16:47:50,086 INFO org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed

Thank you for the feedback Raghav! Your explanation makes complete sense to me. Would you be open to change the time from "number of seconds after start" to normal timestamp? As Hadoop won't be starting all tasks at the same time, it will be hard to correlate multiple mappers without having timestamps (assuming that the time itself will be sync across Hadoop cluster).
- Jarek
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13035/#review25193
On July 30, 2013, 7:37 p.m., Raghav Gautam wrote: