-
Re: ResourceManager not getting started in 0.23 - solvedPraveen Sripati 2011-12-22, 14:21
Patrick,
Thanks for the help.
The historyserver was looking for the hdfs jar files, so I had to add a
similar section for hdfs also in the yarn script. Now all the process start
normally and I am able to run sample program also.
Praveen
On Thu, Dec 22, 2011 at 12:29 PM, Patrick Wendell <[EMAIL PROTECTED]>wrote:
> I had the same problem. Solved by re-adding the following lines back
> to bin/yarn right after CLASSPATH is first declared:
>
> >>>>
> # add hadoop-common libs to CLASSPATH
> if [ ! -d "$HADOOP_COMMON_HOME" ]; then
> echo No HADOOP_COMMON_HOME set.
> echo Please specify it either in yarn-env.sh or in the environment.
> exit 1
> fi
>
> CLASSPATH=${CLASSPATH}:$HADOOP_COMMON_HOME/share/hadoop/common'/*'
> CLASSPATH=${CLASSPATH}:$HADOOP_COMMON_HOME/share/hadoop/common/lib'/*'
> >>>
>
> I've done a clean and re-build and I still need to explicitly include
> those hadoop common JARS or else the RM won't start. Not sure what is
> actually the problem here.
>
> - Patrick
>
> On Wed, Dec 21, 2011 at 8:25 PM, Praveen Sripati
> <[EMAIL PROTECTED]> wrote:
> > Arun,
> >
> > Thanks for the response.
> >
> > I had to explicitly add the hadoop-common-0.23.1-SNAPSHOT.jar in the
> > classpath using the below commands.
> >
> > cd $HADOOP_COMMON_HOME/share/hadoop/mapreduce/
> > ln -s ../common/hadoop-common-0.23.1-SNAPSHOT.jar
> >
> > But, now ResourceManager fails silently with the following output in the
> > yarn-praveensripati-resourcemanager-praveen-laptop.out. When I run jps, I
> > don't see ResourceManager.
> >
> > /usr/lib/jvm/jdk1.6.0_27/bin/java -Dproc_resourcemanager -Xmx1000m
> >
> -Dhadoop.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> >
> -Dyarn.logs.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> > -Dhadoop.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> > -Dyarn.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> > -Dyarn.home.dir= -Dyarn.id.str=praveensripati
> > -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> > -Dyarn.policy.file=hadoop-policy.xml
> >
> -Dhadoop.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> >
> -Dyarn.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> > -Dhadoop.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> > -Dyarn.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> >
> -Dyarn.home.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT
> > -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -classpath
> >
> /home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/share/hadoop/mapreduce/*:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/share/hadoop/mapreduce/lib/*:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/rm-config/log4j.properties
> > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
> > log4j:WARN No appenders could be found for logger
> > (org.apache.hadoop.yarn.server.resourcemanager.ResourceManager).
> > log4j:WARN Please initialize the log4j system properly.
> >
> > Regards,
> > Praveen
> >
> > On Wed, Dec 21, 2011 at 11:08 PM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
> >
> >> Two ways around:
> >>
> >> # You might have a stale version of hadoop-common, apparently that
> should
> >> fix it.
> >> # Upgrade to include MAPREDUCE-3588 which got committed yesterday.
> >>
> >> Arun
> >>
> >>
> >> On Dec 21, 2011, at 9:34 AM, Praveen Sripati wrote:
> >>
> >> > Hi,
> >> >
> >> > I am making some change to the 0.23 branch and got the latest code and
> >> > successfully build it. Am able to start the NameNode and the DataNode,
> >> but
> >> > when I start the ResourceManager I am getting
Thanks for the help.
The historyserver was looking for the hdfs jar files, so I had to add a
similar section for hdfs also in the yarn script. Now all the process start
normally and I am able to run sample program also.
Praveen
On Thu, Dec 22, 2011 at 12:29 PM, Patrick Wendell <[EMAIL PROTECTED]>wrote:
> I had the same problem. Solved by re-adding the following lines back
> to bin/yarn right after CLASSPATH is first declared:
>
> >>>>
> # add hadoop-common libs to CLASSPATH
> if [ ! -d "$HADOOP_COMMON_HOME" ]; then
> echo No HADOOP_COMMON_HOME set.
> echo Please specify it either in yarn-env.sh or in the environment.
> exit 1
> fi
>
> CLASSPATH=${CLASSPATH}:$HADOOP_COMMON_HOME/share/hadoop/common'/*'
> CLASSPATH=${CLASSPATH}:$HADOOP_COMMON_HOME/share/hadoop/common/lib'/*'
> >>>
>
> I've done a clean and re-build and I still need to explicitly include
> those hadoop common JARS or else the RM won't start. Not sure what is
> actually the problem here.
>
> - Patrick
>
> On Wed, Dec 21, 2011 at 8:25 PM, Praveen Sripati
> <[EMAIL PROTECTED]> wrote:
> > Arun,
> >
> > Thanks for the response.
> >
> > I had to explicitly add the hadoop-common-0.23.1-SNAPSHOT.jar in the
> > classpath using the below commands.
> >
> > cd $HADOOP_COMMON_HOME/share/hadoop/mapreduce/
> > ln -s ../common/hadoop-common-0.23.1-SNAPSHOT.jar
> >
> > But, now ResourceManager fails silently with the following output in the
> > yarn-praveensripati-resourcemanager-praveen-laptop.out. When I run jps, I
> > don't see ResourceManager.
> >
> > /usr/lib/jvm/jdk1.6.0_27/bin/java -Dproc_resourcemanager -Xmx1000m
> >
> -Dhadoop.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> >
> -Dyarn.logs.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> > -Dhadoop.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> > -Dyarn.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> > -Dyarn.home.dir= -Dyarn.id.str=praveensripati
> > -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> > -Dyarn.policy.file=hadoop-policy.xml
> >
> -Dhadoop.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> >
> -Dyarn.log.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/logs
> > -Dhadoop.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> > -Dyarn.log.file=yarn-praveensripati-resourcemanager-praveen-laptop.log
> >
> -Dyarn.home.dir=/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT
> > -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -classpath
> >
> /home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/usr/lib/jvm/jdk1.6.0_27/lib/tools.jar:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/share/hadoop/mapreduce/*:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/share/hadoop/mapreduce/lib/*:/home/praveensripati/Hadoop/0.23/deploy/hadoop-0.23.1-SNAPSHOT/conf/rm-config/log4j.properties
> > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
> > log4j:WARN No appenders could be found for logger
> > (org.apache.hadoop.yarn.server.resourcemanager.ResourceManager).
> > log4j:WARN Please initialize the log4j system properly.
> >
> > Regards,
> > Praveen
> >
> > On Wed, Dec 21, 2011 at 11:08 PM, Arun C Murthy <[EMAIL PROTECTED]>
> wrote:
> >
> >> Two ways around:
> >>
> >> # You might have a stale version of hadoop-common, apparently that
> should
> >> fix it.
> >> # Upgrade to include MAPREDUCE-3588 which got committed yesterday.
> >>
> >> Arun
> >>
> >>
> >> On Dec 21, 2011, at 9:34 AM, Praveen Sripati wrote:
> >>
> >> > Hi,
> >> >
> >> > I am making some change to the 0.23 branch and got the latest code and
> >> > successfully build it. Am able to start the NameNode and the DataNode,
> >> but
> >> > when I start the ResourceManager I am getting
