Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Classic(MapReduce 1) cluster in Hadoop 0.23 just won't listen


+
Alexander Hristov 2012-10-03, 04:43
+
Harsh J 2012-10-03, 05:39
+
Alexander Hristov 2012-10-03, 06:06
Copy link to this message
-
Re: Classic(MapReduce 1) cluster in Hadoop 0.23 just won't listen
Hi,

The classic option exists to provide backward compatibility for users
wanting to run an MR1 cluster (with JT, etc.).

With the inclusion of YARN and MR2 modes of runtime, Apache Hadoop
removed MR1 services support:

"""
➜  mapred jobtracker
Sorry, the jobtracker command is no longer supported.
"""

So if you need MR1 running with few hassles, you'll have to either use
the independent version of classic MR from an Apache Hadoop 0.22
release (which had independent components, making this easy) and set
it up as an independent cluster with 0.23 HDFS jars (API is compatible
so should work), or use the MR1 tarball CDH4 offers (which is closer
to 1.x MR1 feature-set), and strip out the CDH4 HDFS jars and then run
the daemons via start-mapred.sh in either of them.

If you instead want to use Apache Hadoop 1.x, you'll need to remove
all HDFS references in the core-jar or exclude it from a build, and
produce an MR-only deployable set. This is harder work to do.

P.s. I haven't tried this out personally, but feel this may work.

On Wed, Oct 3, 2012 at 11:36 AM, Alexander Hristov <[EMAIL PROTECTED]> wrote:
> Thanks for replying.
>
> I'm using the 0.23.3 release as distributed, no previous versions.
>
> So what's the point in documenting a classic option, then, if it is not
> available? I thought distributions were self-contained, or at least the docs
> don't mention that you need any previous versions.
>
>
>
>> What is your 'classic' MapReduce bundle version? 0.23 ships no classic
>> MapReduce services bundle in it AFAIK, only YARN+(MR2-App).
>>
>> Whatever version you're trying to use, make sure it is not using the
>> older HDFS jars?
>>
>> On Wed, Oct 3, 2012 at 10:13 AM, Alexander Hristov <[EMAIL PROTECTED]>
>> wrote:
>>>
>>> Hi again
>>>
>>> Why does it seem to me that everything Hadoop 0.23-related is an uphill
>>> battle? :-(
>>>
>>> I'm trying something as simple as running a classic(MapReduce 1) Hadoop
>>> cluster. Here's my configuration:
>>>
>>> core-site.xml:
>>> <configuration>
>>>      <property>
>>>        <name>fs.default.name</name>
>>>        <value>hdfs://samplehost.com:9000</value>
>>>      </property>
>>> </configuration>
>>>
>>> hdfs-site.xml
>>> <configuration>
>>>       <property>
>>>           <name>dfs.replication</name>
>>>           <value>3</value>
>>>       </property>
>>> </configuration>
>>>
>>> mapred-site.xml
>>> <configuration>
>>>    <property>
>>>      <name>mapreduce.framework.name</name>
>>>      <value>classic</value>
>>>    </property>
>>>    <property>
>>>      <name>mapred.job.tracker</name>
>>>      <value>samplehost.com:9001</value>
>>>    </property>
>>>      <property>
>>>      <name> mapreduce.jobtracker.address</name>
>>>      <value>samplehost.com:9001</value>
>>>    </property>
>>> </configuration>
>>>
>>> yarn-site.xml
>>> <configuration>
>>> </configuration>
>>>
>>>
>>> Well, I start the thing and do a netstat -l , and there's no one
>>> listening
>>> on port 9001. There are no errors in the logs, and no mention of that
>>> port,
>>> either.
>>> Obviously, all Map/Reduce examples fail with Connection Refused.
>>>
>>> Starting the same cluster using a MapReduce 2 (YARN) configuration works
>>> properly.
>>>
>>> Regards,
>>>
>>> Alexander
>>>
>>>
>>>
>>
>>
>

--
Harsh J
+
Harsh J 2012-10-03, 21:11
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB