Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - jobtracker / hadoop comsumer


Copy link to this message
-
Re: jobtracker / hadoop comsumer
Richard Park 2011-08-31, 00:37
I think some changes will be needed with the instructions.

If you have a local Hadoop installed then it goes to the local job tracker.
You'll have to set up your Hadoop to point to the correct cluster. There are
several ways to do it, but you can set your HADOOP_CONF_DIR to point to the
correct hadoop conf xml (ie. location of mapred-site.xml) for the remote
clusters.

The hadoop map tasks will need to connect to the kafka server port (the
broker uri/port).
On Tue, Aug 30, 2011 at 4:50 PM, Ben Ciceron <[EMAIL PROTECTED]> wrote:

> thx for confirming.
>
> so when follow the instructions to run the hadoop consumer
> (https://github.com/kafka-dev/kafka/tree/master/contrib/hadoop-consumer)
> i see my mapred job being submitted properly on hostB (jobtracker) but
> it always fails with :
>
> console output on hostA
>
> 11/08/31 07:32:22 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> 11/08/31 07:32:35 INFO mapred.FileInputFormat: Total input paths to process
> : 1
> Hadoop job id=job_201108291829_0041
> Exception in thread "main" java.lang.Exception: Hadoop ETL job failed!
> Please check status on
> http://localhost:9001/jobdetails.jsp?jobid=job_201108291829_0041
>        at
> kafka.etl.impl.SimpleKafkaETLJob.execute(SimpleKafkaETLJob.java:82)
>        at kafka.etl.impl.SimpleKafkaETLJob.main(SimpleKafkaETLJob.java:100)
>
> in mapred log, for each tasktracker host i see :
>
> Meta VERSION="1" .
> Job JOBID="job_201108291829_0041" JOBNAME="SimpleKafakETL" USER="root"
> SUBMIT_TIME="1314747158794"
>
> JOBCONF="maprfs://10\.18\.125\.176:7222/var/mapr/cluster/mapred/jobTracker/staging/root/\.staging/job_201108291829_0041/job\.xml"
> VIEW_JOB="*" MODIFY_JOB="*" JOB_QUEUE="default" .
> Job JOBID="job_201108291829_0041" JOB_PRIORITY="NORMAL" .
> Job JOBID="job_201108291829_0041" JOB_STATUS="RUNNING" .
> Job JOBID="job_201108291829_0041" LAUNCH_TIME="1314747158885"
> TOTAL_MAPS="1" TOTAL_REDUCES="0" JOB_STATUS="PREP" .
> Task TASKID="task_201108291829_0041_m_000000" TASK_TYPE="MAP"
> START_TIME="1314747160010"
> SPLITS="/default-rack/hadoop2,/default-rack/hadoop9,/default-rack/hadoop6"
> .
> MapAttempt TASK_TYPE="MAP" TASKID="task_201108291829_0041_m_000000"
> TASK_ATTEMPT_ID="attempt_201108291829_0041_m_000000_0"
> START_TIME="1314747160121"
> TRACKER_NAME="tracker_hadoop9:localhost/127\.0\.0\.1:59411"
> HTTP_PORT="50060" .
> MapAttempt TASK_TYPE="MAP" TASKID="task_201108291829_0041_m_000000"
> TASK_ATTEMPT_ID="attempt_201108291829_0041_m_000000_0"
> TASK_STATUS="FAILED" FINISH_TIME="1314747164349" HOSTNAME="hadoop9"
> ERROR="java\.io\.IOException: java\.net\.ConnectException: Connection
> refused
>        at
> kafka\.etl\.KafkaETLRecordReader\.next(KafkaETLRecordReader\.java:155)
>        at
> kafka\.etl\.KafkaETLRecordReader\.next(KafkaETLRecordReader\.java:14)
>        at
> org\.apache\.hadoop\.mapred\.MapTask$TrackedRecordReader\.moveToNext(MapTask\.java:210)
>        at
> org\.apache\.hadoop\.mapred\.MapTask$TrackedRecordReader\.next(MapTask\.java:195)
>        at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:48)
>        at
> org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:393)
>        at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:326)
>        at org\.apache\.hadoop\.mapred\.Child$4\.run(Child\.java:268)
>        at java\.security\.AccessController\.doPrivileged(Native Method)
>        at javax\.security\.auth\.Subject\.doAs(Subject\.java:396)
>        at
> org\.apache\.hadoop\.security\.UserGroupInformation\.doAs(UserGroupInformation\.java:1074)
>        at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:262)
> Caused by: java\.net\.ConnectException: Connection refused
>        at sun\.nio\.ch\.Net\.connect(Native Method)
>        at
> sun\.nio\.ch\.SocketChannelImpl\.connect(SocketChannelImpl\.java:500)
>        at
> kafka\.consumer\.SimpleConsumer\.connect(SimpleConsumer\.scala:54)
>        at
> kafka\.consumer\.SimpleConsumer\.getOrMakeConnection(SimpleConsumer\.scala:193)