Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: Access and archive Job Tracker logs


Copy link to this message
-
Re: Access and archive Job Tracker logs
I digged a little bit deeper and found an interesting "*mapreduce/history"
folder on the JobTracker.

Could you help me how to use these files?
I need to archive them and search them later (by date or job-name).
michaela 18:29:31 /var/log/hadoop-0.20-mapreduce/history # tree
.
├── done
│   ├── michaela.ixcloud.net_1361182032467_
│   │   └── 2013
│   │       └── 01
│   │           └── 18
│   ├── michaela.ixcloud.net_1361195731054_
│   │   └── 2013
│   │       └── 01
│   │           └── 19
│   ├── michaela.ixcloud.net_1363254634813_
│   │   └── 2013
│   │       └── 03
│   │           └── 14
│   │               └── 000000
│   │                   ├──
job_201303141050_0001_1363254715201_christian_rep120.case3ii
│   │                   ├──
job_201303141050_0002_1363254735631_christian_rep120.case3ii
│   │                   ├──
job_201303141050_0003_1363254752878_christian_rep120.case3ii
│   │                   ├──
job_201303141050_0004_1363254766964_christian_rep120.case3ii
│   │                   ├──
job_201303141050_0005_1363254779720_christian_rep120.case3ii
│   │                   ├──
job_201303141050_0006_1363254851856_panshul_PigLatin%3Agametypecounter
│   │                   ├──
job_201303141050_0007_1363255612659_lealem_gameDataTimeSeriesJob
│   │                   ├──
job_201303141050_0010_1363263699635_lealem_gameDataTimeSeriesJob
│   │                   ├──
michaela.ixcloud.net_1363254634813_job_201303141050_0001_conf.xml
│   │                   ├──
michaela.ixcloud.net_1363254634813_job_201303141050_0002_conf.xml
│   │                   ├──
michaela.ixcloud.net_1363254634813_job_201303141050_0003_conf.xml
│   │                   ├──
michaela.ixcloud.net_1363254634813_job_201303141050_0004_conf.xml
│   │                   ├──
michaela.ixcloud.net_1363254634813_job_201303141050_0005_conf.xml
│   │                   ├──
michaela.ixcloud.net_1363254634813_job_201303141050_0006_conf.xml
│   │                   ├──
michaela.ixcloud.net_1363254634813_job_201303141050_0007_conf.xml
│   │                   └──
michaela.ixcloud.net_1363254634813_job_201303141050_0010_conf.xml
│   └── michaela.ixcloud.net_1363269263476_
│       └── 2013
│           └── 03
│               └── 14
│                   └── 000000
│                       ├──
job_201303141454_0001_1363269701945_root_importtsv_timeSeries
│                       ├──
job_201303141454_0002_1363269757097_hdfs_importtsv_timeSeries
│                       ├──
job_201303141454_0003_1363271517403_christian_rep120.case3i
│                       ├──
job_201303141454_0005_1363280494548_lealem_gameDataTimeSeriesJob
│                       ├──
michaela.ixcloud.net_1363269263476_job_201303141454_0001_conf.xml
│                       ├──
michaela.ixcloud.net_1363269263476_job_201303141454_0002_conf.xml
│                       ├──
michaela.ixcloud.net_1363269263476_job_201303141454_0003_conf.xml
│                       └──
michaela.ixcloud.net_1363269263476_job_201303141454_0005_conf.xml
├──
job_201303061254_0010_1362574214308_tim_SELECT+*+FROM+temp_panshul_table+WHERE+...10%28Stage
├── job_201303061458_0015_1362582264552_tim_hivetest42
├── job_201303061737_0006_1362588850460_christian_rankingJob
├── job_201303141454_0006_1363281646038_lars+job+2009
├── michaela.ixcloud.net_1362570865029_job_201303061254_0010_conf.xml
├── michaela.ixcloud.net_1362578315853_job_201303061458_0015_conf.xml
├── michaela.ixcloud.net_1362587825736_job_201303061737_0006_conf.xml
└── michaela.ixcloud.net_1363269263476_job_201303141454_0006_conf.xml

I found some information [1] and [2] but those are from 2009-10.
And with the hadoop job -history [3] command i am also not able to read
them (because some _logs/history) folder is missing.

Best Regards,
Christian.

[1]
http://blog.cloudera.com/blog/2009/09/apache-hadoop-log-files-where-to-find-them-in-cdh-and-what-info-they-contain/
[2] http://blog.cloudera.com/blog/2010/11/hadoop-log-location-and-retention/
[3] http://archive.cloudera.com/cdh4/cdh/4/mr1/cluster_setup.html#Logging
2013/3/13 Christian Schneider <[EMAIL PROTECTED]>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB