Dear all,

I got a weird problem when programming on the pseudo-distributed mode of
HBase/Hadoop.

The HBase/Hadoop were installed correctly. It also ran well with my Java
code.

However, if after shutting down the server for some time, for example, four
or five days, I noticed that HBase/Hadoop got a problem. I got an ERROR
when typing "status" in the shell of HBase.

    ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried 7
times

Such a problem had happened for three times in the three weeks.

The HBase/Hadoop are installed on Ubuntu 10.

Have you encountered such a problem? How to solve it?

Thanks so much!

Best regards,
Bing

Hi Bing,

Two questions:

- Can you look at the master log and see what's preventing the master
from starting?

- Did you change dfs.data.dir and dfs.name.dir in hdfs-site.xml? By
default it writes to /tmp which can get cleaned up.

J-D

On Tue, Mar 27, 2012 at 12:52 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear all,
>
> I got a weird problem when programming on the pseudo-distributed mode of
> HBase/Hadoop.
>
> The HBase/Hadoop were installed correctly. It also ran well with my Java
> code.
>
> However, if after shutting down the server for some time, for example, four
> or five days, I noticed that HBase/Hadoop got a problem. I got an ERROR
> when typing "status" in the shell of HBase.
>
>    ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried 7
> times
>
> Such a problem had happened for three times in the three weeks.
>
> The HBase/Hadoop are installed on Ubuntu 10.
>
> Have you encountered such a problem? How to solve it?
>
> Thanks so much!
>
> Best regards,
> Bing

It says you have not started the hbase master. Once you restarted the system have you confirmed whether all hadoop daemons are running?
sudo jps
If you are using CDH package then you can automatically start the hadoop daemons on boot using reconfig package.

Sent from my BlackBerry, pls excuse typo

-----Original Message-----
From: Bing Li <[EMAIL PROTECTED]>
Date: Wed, 28 Mar 2012 03:52:12
To: <[EMAIL PROTECTED]>; user<[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Starting Abnormally After Shutting Down For Some Time

Dear all,

I got a weird problem when programming on the pseudo-distributed mode of
HBase/Hadoop.

The HBase/Hadoop were installed correctly. It also ran well with my Java
code.

However, if after shutting down the server for some time, for example, four
or five days, I noticed that HBase/Hadoop got a problem. I got an ERROR
when typing "status" in the shell of HBase.

    ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried 7
times

Such a problem had happened for three times in the three weeks.

The HBase/Hadoop are installed on Ubuntu 10.

Have you encountered such a problem? How to solve it?

Thanks so much!

Best regards,
Bing

Dear Manish and Jean-Daniel,

After starting DFS (/opt/hadoop/bin/start-dfs.sh), I got the following
daemons after tying "jps".

5212 Jps
5150 SecondaryNameNode
4932 DataNode
4737 NameNode

Then, I started the HBase (/opt/hbase/bin/start-hbase.sh). The following
daemons were available.

5797 Jps
5526 HMaster
5150 SecondaryNameNode
5711 HRegionServer
4932 DataNode
4737 NameNode
5456 HQuorumPeer

HMaster was started. It seemed that everything was fine.

But when typing "status" in HBase shell. The following error still occurred.

ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried 7 times

In the master log, the following exception was found.

2012-03-28 13:40:01,193 FATAL org.apache.hadoop.hbase.master.HMaster:
Unhandled exception. Starting shutdown.
java.net.ConnectException: Call to localhost/127.0.0.1:9000 failed on
connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1095)
at org.apache.hadoop.ipc.Client.call(Client.java:1071)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy10.setSafeMode(Unknown Source)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy10.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:1120)
at
org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:423)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:439)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:323)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:128)
at
org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:113)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:448)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:326)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:656)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
at org.apache.hadoop.ipc.Client.call(Client.java:1046)
... 17 more
2012-03-28 13:40:01,195 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting

What is the problem? Why does it happen after HBase/Hadoop is shutdown for
a couple of days?

Thanks so much!

Bing

On Wed, Mar 28, 2012 at 11:09 AM, Manish Bhoge
<[EMAIL PROTECTED]>wrote:

> It says you have not started the hbase master. Once you restarted the
> system have you confirmed whether all hadoop daemons are running?
> sudo jps
> If you are using CDH package then you can automatically start the hadoop
> daemons on boot using reconfig package.
>
> Sent from my BlackBerry, pls excuse typo
>
> -----Original Message-----
> From: Bing Li <[EMAIL PROTECTED]>
> Date: Wed, 28 Mar 2012 03:52:12
> To: <[EMAIL PROTECTED]>; user<[EMAIL PROTECTED]>
> Reply-To: [EMAIL PROTECTED]
> Subject: Starting Abnormally After Shutting Down For Some Time
>
> Dear all,
>
> I got a weird problem when programming on the pseudo-distributed mode of
> HBase/Hadoop.
>
> The HBase/Hadoop were installed correctly. It also ran well with my Java
> code.

Jean-Daniel,

I changed dfs.data.dir and dfs.name.dir to new paths in the hdfs-site.xml.

I really cannot figure out why the HBase/Hadoop got a problem after a
couple of days of shutting down. If I use it frequently, no such a master
problem happens.

Each time, I have to reinstall not only HBase/Hadoop but also Ubuntu for
the problem. It wasted me a lot of time.

Thanks so much!

Bing

On Wed, Mar 28, 2012 at 4:46 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> Hi Bing,
>
> Two questions:
>
> - Can you look at the master log and see what's preventing the master
> from starting?
>
> - Did you change dfs.data.dir and dfs.name.dir in hdfs-site.xml? By
> default it writes to /tmp which can get cleaned up.
>
> J-D
>
> On Tue, Mar 27, 2012 at 12:52 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > Dear all,
> >
> > I got a weird problem when programming on the pseudo-distributed mode of
> > HBase/Hadoop.
> >
> > The HBase/Hadoop were installed correctly. It also ran well with my Java
> > code.
> >
> > However, if after shutting down the server for some time, for example,
> four
> > or five days, I noticed that HBase/Hadoop got a problem. I got an ERROR
> > when typing "status" in the shell of HBase.
> >
> >    ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried 7
> > times
> >
> > Such a problem had happened for three times in the three weeks.
> >
> > The HBase/Hadoop are installed on Ubuntu 10.
> >
> > Have you encountered such a problem? How to solve it?
> >
> > Thanks so much!
> >
> > Best regards,
> > Bing
>

Bing,

As per my experience on the configuration I can list down some points one of which may be your solution.

- first and foremost don't store your service metadata into system tmp directory because it may get cleaned up in every start and you loose all your job tracker, datanode information. It is as good as you're formatting your namenode.
- if you're using CDH make sure you set up permission perfectly for root, dfs data directory and mapred directories.(Refer CDH documentation)
- I didn't see job tracker in your service list. It should be up and running. Check the job tracker log if there is any permission issue when starting job tracker and task tracker.
- before trying your stuff on Hbase set up make sure all your Hadoop services are up and running. You can check that by running a sample program and check whether job tracker, task tracker responding for your mapred.system and mapred.local directories to create intermediate files.
- once you have all hadoop services up don't set/change any permission.

As far as Hbase configuration is concerned there are 2 path for set up: either you set up zookeeper within hbase-site.xml Or configure separately via zoo.cfg. If you are going with hbase setting for zookeeper then confirm following setting:
- hbase.zookeeper.property.clientPort.
- hbase.zookeeper.quorum.
- hbase.zookeeper.property.dataDir
Once you have right setting for these and set up root directory for hbase then there not much excercise is required.(Make sure zookeeper service is up before you start hbase)

I think if you follow above rules you should be fine. There is no issue because of long time shutdown or frequent machine restart.

 Champ, moreover you need to have good amount of patience to understand the problem :) I do understand how frustating when you set up everything and next day you find the things are completely down.

Sent from my BlackBerry, pls excuse typo

-----Original Message-----
From: Bing Li <[EMAIL PROTECTED]>
Date: Wed, 28 Mar 2012 14:32:12
To: <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: Starting Abnormally After Shutting Down For Some Time

Jean-Daniel,

I changed dfs.data.dir and dfs.name.dir to new paths in the hdfs-site.xml.

I really cannot figure out why the HBase/Hadoop got a problem after a
couple of days of shutting down. If I use it frequently, no such a master
problem happens.

Each time, I have to reinstall not only HBase/Hadoop but also Ubuntu for
the problem. It wasted me a lot of time.

Thanks so much!

Bing

On Wed, Mar 28, 2012 at 4:46 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> Hi Bing,
>
> Two questions:
>
> - Can you look at the master log and see what's preventing the master
> from starting?
>
> - Did you change dfs.data.dir and dfs.name.dir in hdfs-site.xml? By
> default it writes to /tmp which can get cleaned up.
>
> J-D
>
> On Tue, Mar 27, 2012 at 12:52 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > Dear all,
> >
> > I got a weird problem when programming on the pseudo-distributed mode of
> > HBase/Hadoop.
> >
> > The HBase/Hadoop were installed correctly. It also ran well with my Java
> > code.
> >
> > However, if after shutting down the server for some time, for example,
> four
> > or five days, I noticed that HBase/Hadoop got a problem. I got an ERROR
> > when typing "status" in the shell of HBase.
> >
> >    ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried 7
> > times
> >
> > Such a problem had happened for three times in the three weeks.
> >
> > The HBase/Hadoop are installed on Ubuntu 10.
> >
> > Have you encountered such a problem? How to solve it?
> >
> > Thanks so much!
> >
> > Best regards,
> > Bing
>

Dear Manish,

I appreciate so much for your replies!

The system tmp directory is changed to anther location in my hdfs-site.xml.

If I ran $HADOOP_HOME/bin/start-all.sh, all of the services were listed,
including job tracker and task tracker.

    10211 SecondaryNameNode
    10634 Jps
    9992 DataNode
    10508 TaskTracker
    10312 JobTracker
    9797 NameNode

In the job tracker's log, one exception was found.

       org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/home/libing/GreatFreeLab
s/Hadoop/FS/mapred/system. Name node is in safe mode.

In my system, I didn't see the directory, ~/mapred. How should I configure
for it?

For the properties you listed, they were not set in my system. Are they
required? Since they have default values (
http://hbase.apache.org/docs/r0.20.6/hbase-conf.html), do I need to update
them?

     - hbase.zookeeper.property.clientPort.
     - hbase.zookeeper.quorum.
     - hbase.zookeeper.property.dataDir

Now the system was reinstalled. At least, the pseudo-distributed mode runs
well. I also tried to shut down the ubuntu machine and started it again.
The system worked fine. But I worried the master-related problem must
happen if the machine was shutdown for more time. I really don't understand
the reason.

Thanks so much!

Best,
Bing

On Wed, Mar 28, 2012 at 3:11 PM, Manish Bhoge <[EMAIL PROTECTED]>wrote:

> Bing,
>
> As per my experience on the configuration I can list down some points one
> of which may be your solution.
>
> - first and foremost don't store your service metadata into system tmp
> directory because it may get cleaned up in every start and you loose all
> your job tracker, datanode information. It is as good as you're formatting
> your namenode.
> - if you're using CDH make sure you set up permission perfectly for root,
> dfs data directory and mapred directories.(Refer CDH documentation)
> - I didn't see job tracker in your service list. It should be up and
> running. Check the job tracker log if there is any permission issue when
> starting job tracker and task tracker.
> - before trying your stuff on Hbase set up make sure all your Hadoop
> services are up and running. You can check that by running a sample program
> and check whether job tracker, task tracker responding for your
> mapred.system and mapred.local directories to create intermediate files.
> - once you have all hadoop services up don't set/change any permission.
>
> As far as Hbase configuration is concerned there are 2 path for set up:
> either you set up zookeeper within hbase-site.xml Or configure separately
> via zoo.cfg. If you are going with hbase setting for zookeeper then confirm
> following setting:
> - hbase.zookeeper.property.clientPort.
> - hbase.zookeeper.quorum.
> - hbase.zookeeper.property.dataDir
> Once you have right setting for these and set up root directory for hbase
> then there not much excercise is required.(Make sure zookeeper service is
> up before you start hbase)
>
> I think if you follow above rules you should be fine. There is no issue
> because of long time shutdown or frequent machine restart.
>
>  Champ, moreover you need to have good amount of patience to understand
> the problem :) I do understand how frustating when you set up everything
> and next day you find the things are completely down.
>
> Sent from my BlackBerry, pls excuse typo
>
> -----Original Message-----
> From: Bing Li <[EMAIL PROTECTED]>
> Date: Wed, 28 Mar 2012 14:32:12
> To: <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]>
> Reply-To: [EMAIL PROTECTED]
> Subject: Re: Starting Abnormally After Shutting Down For Some Time
>
> Jean-Daniel,
>
> I changed dfs.data.dir and dfs.name.dir to new paths in the hdfs-site.xml.
>
> I really cannot figure out why the HBase/Hadoop got a problem after a
> couple of days of shutting down. If I use it frequently, no such a master
> problem happens.
>
> Each time, I have to reinstall not only HBase/Hadoop but also Ubuntu for

R

----- Original Message -----
From: Bing Li [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, March 28, 2012 01:32 AM
To: [EMAIL PROTECTED] <[EMAIL PROTECTED]>; [EMAIL PROTECTED] <[EMAIL PROTECTED]>
Subject: Re: Starting Abnormally After Shutting Down For Some Time

Jean-Daniel,

I changed dfs.data.dir and dfs.name.dir to new paths in the hdfs-site.xml.

I really cannot figure out why the HBase/Hadoop got a problem after a
couple of days of shutting down. If I use it frequently, no such a master
problem happens.

Each time, I have to reinstall not only HBase/Hadoop but also Ubuntu for
the problem. It wasted me a lot of time.

Thanks so much!

Bing

On Wed, Mar 28, 2012 at 4:46 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> Hi Bing,
>
> Two questions:
>
> - Can you look at the master log and see what's preventing the master
> from starting?
>
> - Did you change dfs.data.dir and dfs.name.dir in hdfs-site.xml? By
> default it writes to /tmp which can get cleaned up.
>
> J-D
>
> On Tue, Mar 27, 2012 at 12:52 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > Dear all,
> >
> > I got a weird problem when programming on the pseudo-distributed mode of
> > HBase/Hadoop.
> >
> > The HBase/Hadoop were installed correctly. It also ran well with my Java
> > code.
> >
> > However, if after shutting down the server for some time, for example,
> four
> > or five days, I noticed that HBase/Hadoop got a problem. I got an ERROR
> > when typing "status" in the shell of HBase.
> >
> >    ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried 7
> > times
> >
> > Such a problem had happened for three times in the three weeks.
> >
> > The HBase/Hadoop are installed on Ubuntu 10.
> >
> > Have you encountered such a problem? How to solve it?
> >
> > Thanks so much!
> >
> > Best regards,
> > Bing
>

Dear all,

I found some configuration information was saved in /tmp in my system. So
when some of the information is lost, the HBase cannot be started normally.

But in my system, I have tried to change the HDFS directory to another
location. Why are there still some files under /tmp?

To change the HDFS directory, the hdfs-site.xml is updated as follows. What
else should I do for moving all the configurations out of /tmp?

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
      <property>
        <name>dfs.replication</name>
        <value>1</value>
      </property>
      <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/libing/GreatFreeLabs/Hadoop/FS</value>
      </property>
      <property>
        <name>dfs.name.dir</name>
        <value>${hadoop.tmp.dir}/dfs/name/</value>
      </property>
      <property>
        <name>dfs.data.dir</name>
        <value>${hadoop.tmp.dir}/dfs/data/</value>
      </property>
   </configuration>

Thanks so much!

Best,
Bing

On Wed, Mar 28, 2012 at 4:24 PM, Bing Li <[EMAIL PROTECTED]> wrote:

> Dear Manish,
>
> I appreciate so much for your replies!
>
> The system tmp directory is changed to anther location in my hdfs-site.xml.
>
> If I ran $HADOOP_HOME/bin/start-all.sh, all of the services were listed,
> including job tracker and task tracker.
>
>     10211 SecondaryNameNode
>     10634 Jps
>     9992 DataNode
>     10508 TaskTracker
>     10312 JobTracker
>     9797 NameNode
>
> In the job tracker's log, one exception was found.
>
>        org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> /home/libing/GreatFreeLab
> s/Hadoop/FS/mapred/system. Name node is in safe mode.
>
> In my system, I didn't see the directory, ~/mapred. How should I configure
> for it?
>
> For the properties you listed, they were not set in my system. Are they
> required? Since they have default values (
> http://hbase.apache.org/docs/r0.20.6/hbase-conf.html), do I need to
> update them?
>
>      - hbase.zookeeper.property.clientPort.
>      - hbase.zookeeper.quorum.
>      - hbase.zookeeper.property.dataDir
>
> Now the system was reinstalled. At least, the pseudo-distributed mode runs
> well. I also tried to shut down the ubuntu machine and started it again.
> The system worked fine. But I worried the master-related problem must
> happen if the machine was shutdown for more time. I really don't understand
> the reason.
>
> Thanks so much!
>
> Best,
> Bing
>
> On Wed, Mar 28, 2012 at 3:11 PM, Manish Bhoge <[EMAIL PROTECTED]>wrote:
>
>> Bing,
>>
>> As per my experience on the configuration I can list down some points one
>> of which may be your solution.
>>
>> - first and foremost don't store your service metadata into system tmp
>> directory because it may get cleaned up in every start and you loose all
>> your job tracker, datanode information. It is as good as you're formatting
>> your namenode.
>> - if you're using CDH make sure you set up permission perfectly for root,
>> dfs data directory and mapred directories.(Refer CDH documentation)
>> - I didn't see job tracker in your service list. It should be up and
>> running. Check the job tracker log if there is any permission issue when
>> starting job tracker and task tracker.
>> - before trying your stuff on Hbase set up make sure all your Hadoop
>> services are up and running. You can check that by running a sample program
>> and check whether job tracker, task tracker responding for your
>> mapred.system and mapred.local directories to create intermediate files.
>> - once you have all hadoop services up don't set/change any permission.
>>
>> As far as Hbase configuration is concerned there are 2 path for set up:
>> either you set up zookeeper within hbase-site.xml Or configure separately
>> via zoo.cfg. If you are going with hbase setting for zookeeper then confirm
>> following setting:
>> - hbase.zookeeper.property.clientPort.

On Wed, Mar 28, 2012 at 7:27 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear all,
>
> I found some configuration information was saved in /tmp in my system. So
> when some of the information is lost, the HBase cannot be started normally.
>
> But in my system, I have tried to change the HDFS directory to another
> location. Why are there still some files under /tmp?

I have a pseudo-distributed set-up (Cloudera cdh3u2) with local
directory (not /tmp)
and as a test:

* stopped the hbase service
* stopped the hadoop services
* moved all hadoop related files from tmp to an ORIG directory [see below]
* restarted all (5) hadoop services
* restarted the hbase service

All of that worked stable, so I presume no immediate dependency on the
/tmp files. The files that are recreated are these:

peterv@e6500:/tmp$ ls -alrt
...
drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 20:07
Jetty_0_0_0_0_50070_hdfs____w2cu08
drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 20:07
Jetty_0_0_0_0_50075_datanode____hwtdwq
drwxr-xr-x  2 hdfs   hdfs   4096 2012-03-28 20:07 hsperfdata_hdfs
drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 20:07
Jetty_0_0_0_0_50090_secondary____y6aanv
drwxr-xr-x  4 mapred mapred 4096 2012-03-28 20:07
Jetty_0_0_0_0_50030_job____yn7qmk
drwxr-xr-x  2 mapred mapred 4096 2012-03-28 20:07 hsperfdata_mapred
drwxr-xr-x  2 root   root   4096 2012-03-28 20:07 hsperfdata_root
drwxr-xr-x  4 mapred mapred 4096 2012-03-28 20:07
Jetty_0_0_0_0_50060_task____.2vcltf

The files that I had moved on the site (to ORIG) where:

peterv@e6500:/tmp$ ls -alrt ORIG/
total 44
drwxr-xr-x  4 mapred mapred 4096 2012-03-28 19:58
Jetty_0_0_0_0_50030_job____yn7qmk
drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 19:58
Jetty_0_0_0_0_50070_hdfs____w2cu08
drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 19:58
Jetty_0_0_0_0_50090_secondary____y6aanv
drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 19:58
Jetty_0_0_0_0_50075_datanode____hwtdwq
drwxr-xr-x  4 mapred mapred 4096 2012-03-28 19:59
Jetty_0_0_0_0_50060_task____.2vcltf
drwxr-xr-x  2 peterv peterv 4096 2012-03-28 20:05 hsperfdata_peterv
drwxr-xr-x  2 hdfs   hdfs   4096 2012-03-28 20:05 hsperfdata_hdfs
drwxr-xr-x  2 mapred mapred 4096 2012-03-28 20:05 hsperfdata_mapred
drwxr-xr-x  2 root   root   4096 2012-03-28 20:06 hsperfdata_root

Which hadoop/hbase files do you still see in your /tmp directory?

HTH,

Peter

Dear Peter,

When I just started the Ubuntu machine, there was nothing in /tmp.

After starting $HADOOP/bin/start-dfs.sh and $HBase/bin/start-hbase.sh, the
following files were under /tmp. Do you think anything wrong? Thanks!

libing@greatfreeweb:/tmp$ ls -alrt
total 112
drwxr-xr-x 22 root   root    4096 2012-03-28 14:17 ..
-rw-r--r--  1 libing libing     5 2012-03-29 04:48
hadoop-libing-namenode.pid
-rw-r--r--  1 libing libing     5 2012-03-29 04:48
hadoop-libing-datanode.pid
-rw-r--r--  1 libing libing     5 2012-03-29 04:48
hadoop-libing-secondarynamenode.pid
-rw-r--r--  1 libing libing     5 2012-03-29 04:48
hbase-libing-zookeeper.pid
drwxr-xr-x  3 libing libing  4096 2012-03-29 04:48 hbase-libing
-rw-r--r--  1 libing libing     5 2012-03-29 04:48 hbase-libing-master.pid
-rw-r--r--  1 libing libing     5 2012-03-29 04:48
hbase-libing-regionserver.pid
drwxr-xr-x  2 libing libing  4096 2012-03-29 04:48 hsperfdata_libing
drwxrwxrwt  4 root   root    4096 2012-03-29 04:48 .
-rw-r--r--  1 libing libing 71819 2012-03-29 04:48
jffi5395899026867792565.tmp
libing@greatfreeweb:/tmp$

Best,
Bing

On Thu, Mar 29, 2012 at 3:19 AM, Peter Vandenabeele
<[EMAIL PROTECTED]>wrote:

> On Wed, Mar 28, 2012 at 7:27 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> > Dear all,
> >
> > I found some configuration information was saved in /tmp in my system. So
> > when some of the information is lost, the HBase cannot be started
> normally.
> >
> > But in my system, I have tried to change the HDFS directory to another
> > location. Why are there still some files under /tmp?
>
> I have a pseudo-distributed set-up (Cloudera cdh3u2) with local
> directory (not /tmp)
> and as a test:
>
> * stopped the hbase service
> * stopped the hadoop services
> * moved all hadoop related files from tmp to an ORIG directory [see below]
> * restarted all (5) hadoop services
> * restarted the hbase service
>
> All of that worked stable, so I presume no immediate dependency on the
> /tmp files. The files that are recreated are these:
>
> peterv@e6500:/tmp$ ls -alrt
> ...
> drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 20:07
> Jetty_0_0_0_0_50070_hdfs____w2cu08
> drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 20:07
> Jetty_0_0_0_0_50075_datanode____hwtdwq
> drwxr-xr-x  2 hdfs   hdfs   4096 2012-03-28 20:07 hsperfdata_hdfs
> drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 20:07
> Jetty_0_0_0_0_50090_secondary____y6aanv
> drwxr-xr-x  4 mapred mapred 4096 2012-03-28 20:07
> Jetty_0_0_0_0_50030_job____yn7qmk
> drwxr-xr-x  2 mapred mapred 4096 2012-03-28 20:07 hsperfdata_mapred
> drwxr-xr-x  2 root   root   4096 2012-03-28 20:07 hsperfdata_root
> drwxr-xr-x  4 mapred mapred 4096 2012-03-28 20:07
> Jetty_0_0_0_0_50060_task____.2vcltf
>
> The files that I had moved on the site (to ORIG) where:
>
> peterv@e6500:/tmp$ ls -alrt ORIG/
> total 44
> drwxr-xr-x  4 mapred mapred 4096 2012-03-28 19:58
> Jetty_0_0_0_0_50030_job____yn7qmk
> drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 19:58
> Jetty_0_0_0_0_50070_hdfs____w2cu08
> drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 19:58
> Jetty_0_0_0_0_50090_secondary____y6aanv
> drwxr-xr-x  4 hdfs   hdfs   4096 2012-03-28 19:58
> Jetty_0_0_0_0_50075_datanode____hwtdwq
> drwxr-xr-x  4 mapred mapred 4096 2012-03-28 19:59
> Jetty_0_0_0_0_50060_task____.2vcltf
> drwxr-xr-x  2 peterv peterv 4096 2012-03-28 20:05 hsperfdata_peterv
> drwxr-xr-x  2 hdfs   hdfs   4096 2012-03-28 20:05 hsperfdata_hdfs
> drwxr-xr-x  2 mapred mapred 4096 2012-03-28 20:05 hsperfdata_mapred
> drwxr-xr-x  2 root   root   4096 2012-03-28 20:06 hsperfdata_root
>
> Which hadoop/hbase files do you still see in your /tmp directory?
>
> HTH,
>
> Peter
>

On Wed, Mar 28, 2012 at 9:53 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear Peter,
>
> When I just started the Ubuntu machine, there was nothing in /tmp.
>
> After starting $HADOOP/bin/start-dfs.sh and $HBase/bin/start-hbase.sh, the
> following files were under /tmp. Do you think anything wrong? Thanks!
>
> libing@greatfreeweb:/tmp$ ls -alrt
> total 112
> drwxr-xr-x 22 root   root    4096 2012-03-28 14:17 ..
> -rw-r--r--  1 libing libing     5 2012-03-29 04:48
> hadoop-libing-namenode.pid
> -rw-r--r--  1 libing libing     5 2012-03-29 04:48
> hadoop-libing-datanode.pid
> -rw-r--r--  1 libing libing     5 2012-03-29 04:48
> hadoop-libing-secondarynamenode.pid
> -rw-r--r--  1 libing libing     5 2012-03-29 04:48
> hbase-libing-zookeeper.pid
> drwxr-xr-x  3 libing libing  4096 2012-03-29 04:48 hbase-libing
> -rw-r--r--  1 libing libing     5 2012-03-29 04:48 hbase-libing-master.pid
> -rw-r--r--  1 libing libing     5 2012-03-29 04:48
> hbase-libing-regionserver.pid
> drwxr-xr-x  2 libing libing  4096 2012-03-29 04:48 hsperfdata_libing
> drwxrwxrwt  4 root   root    4096 2012-03-29 04:48 .
> -rw-r--r--  1 libing libing 71819 2012-03-29 04:48
> jffi5395899026867792565.tmp
> libing@greatfreeweb:/tmp$
>
> Best,
> Bing

Hmmm, all these files are owned by user 'libing' ...
that is different from my set-up.

Which manual are you exactly following for the pseudo-distributed
installation? In the Cloudera manual that I followed (cdh3u2) there
was a mention of making different users IIRC.

Also, in my set-up the hadoop is started automatically at boot-up
from the scripts in

/etc/rc2.d/S20hadoop-...

where user root then performs an su to user
=> hdfs for the Name, Secondaryname, Data node
=> mapred for the Jobtracker and Tasktracker.

I am not sure it is actually the intention that you start the
5 hadoop processes with the

"... $HADOOP/bin/start-dfs.sh ..."

command as you describe.

I stop and start them with

sudo /etc/init.d/hadoop-0.20-datanode {stop|start}
sudo /etc/init.d/hadoop-0.20-namenode {stop|start}
sudo /etc/init.d/hadoop-0.20-secondarynamenode {stop|start}
sudo /etc/init.d/hadoop-0.20-tasktracker {stop|start}
sudo /etc/init.d/hadoop-0.20-jobtracker {stop|start}

and this seems to be stable for now :-)

But maybe the manual that you follow gives other advise ?

HTH (not sure, I am a beginner too ...)

Peter

Bing:
Your pid file location can be setup via hbase-env.sh; default is /tmp ...

# The directory where pid files are stored. /tmp by default.
# export HBASE_PID_DIR=/var/hadoop/pids
On Wed, Mar 28, 2012 at 3:04 PM, Peter Vandenabeele
<[EMAIL PROTECTED]> wrote:
> On Wed, Mar 28, 2012 at 9:53 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>> Dear Peter,
>>
>> When I just started the Ubuntu machine, there was nothing in /tmp.
>>
>> After starting $HADOOP/bin/start-dfs.sh and $HBase/bin/start-hbase.sh, the
>> following files were under /tmp. Do you think anything wrong? Thanks!
>>
>> libing@greatfreeweb:/tmp$ ls -alrt
>> total 112
>> drwxr-xr-x 22 root   root    4096 2012-03-28 14:17 ..
>> -rw-r--r--  1 libing libing     5 2012-03-29 04:48
>> hadoop-libing-namenode.pid
>> -rw-r--r--  1 libing libing     5 2012-03-29 04:48
>> hadoop-libing-datanode.pid
>> -rw-r--r--  1 libing libing     5 2012-03-29 04:48
>> hadoop-libing-secondarynamenode.pid
>> -rw-r--r--  1 libing libing     5 2012-03-29 04:48
>> hbase-libing-zookeeper.pid
>> drwxr-xr-x  3 libing libing  4096 2012-03-29 04:48 hbase-libing
>> -rw-r--r--  1 libing libing     5 2012-03-29 04:48 hbase-libing-master.pid
>> -rw-r--r--  1 libing libing     5 2012-03-29 04:48
>> hbase-libing-regionserver.pid
>> drwxr-xr-x  2 libing libing  4096 2012-03-29 04:48 hsperfdata_libing
>> drwxrwxrwt  4 root   root    4096 2012-03-29 04:48 .
>> -rw-r--r--  1 libing libing 71819 2012-03-29 04:48
>> jffi5395899026867792565.tmp
>> libing@greatfreeweb:/tmp$
>>
>> Best,
>> Bing
>
> Hmmm, all these files are owned by user 'libing' ...
> that is different from my set-up.
>
> Which manual are you exactly following for the pseudo-distributed
> installation? In the Cloudera manual that I followed (cdh3u2) there
> was a mention of making different users IIRC.
>
> Also, in my set-up the hadoop is started automatically at boot-up
> from the scripts in
>
> /etc/rc2.d/S20hadoop-...
>
> where user root then performs an su to user
> => hdfs for the Name, Secondaryname, Data node
> => mapred for the Jobtracker and Tasktracker.
>
> I am not sure it is actually the intention that you start the
> 5 hadoop processes with the
>
> "... $HADOOP/bin/start-dfs.sh ..."
>
> command as you describe.
>
> I stop and start them with
>
> sudo /etc/init.d/hadoop-0.20-datanode {stop|start}
> sudo /etc/init.d/hadoop-0.20-namenode {stop|start}
> sudo /etc/init.d/hadoop-0.20-secondarynamenode {stop|start}
> sudo /etc/init.d/hadoop-0.20-tasktracker {stop|start}
> sudo /etc/init.d/hadoop-0.20-jobtracker {stop|start}
>
> and this seems to be stable for now :-)
>
> But maybe the manual that you follow gives other advise ?
>
> HTH (not sure, I am a beginner too ...)
>
> Peter