Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Problem with MR job


Copy link to this message
-
Re: Problem with MR job

Hi,

The status seems healthy and the datanodes live:
Status: HEALTHY
  Total size:    118805326 B
  Total dirs:    31
  Total files:    38
  Total blocks (validated):    38 (avg. block size 3126455 B)
  Minimally replicated blocks:    38 (100.0 %)
  Over-replicated blocks:    0 (0.0 %)
  Under-replicated blocks:    9 (23.68421 %)
  Mis-replicated blocks:        0 (0.0 %)
  Default replication factor:    1
  Average block replication:    1.2368422
  Corrupt blocks:        0
  Missing replicas:        72 (153.19148 %)
  Number of data-nodes:        2
  Number of racks:        1
FSCK ended at Wed Sep 21 10:06:17 EDT 2011 in 9 milliseconds
The filesystem under path '/' is HEALTHY

The jps command has the following output:
hdfs@master:~$ jps
24292 SecondaryNameNode
30010 Jps
24109 DataNode
23962 NameNode

Shouldn't this have two datanode listings? In our system, one of the
datanodes and the namenode is the same machine, but i seem to remember
that in the past even with this setup two datanode listings appeared in
the jps output.

Thanks,
George
On 9/21/2011 5:08 PM, Uma Maheswara Rao G 72686 wrote:
> Hi,
>
>   Any cluster restart happend? ..is your NameNode detecting DataNodes as live?
>   Looks DNs did not report anyblocks to NN yet. You have 13 blocks persisted in NameNode namespace. At least 12 blocks should be reported from your DNs. Other wise automatically it will not come out of safemode.
>
> Regards,
> Uma
> ----- Original Message -----
> From: George Kousiouris<[EMAIL PROTECTED]>
> Date: Wednesday, September 21, 2011 7:29 pm
> Subject: Problem with MR job
> To: "[EMAIL PROTECTED]"<[EMAIL PROTECTED]>
>
>> Hi all,
>>
>> We are trying to run a mahout job in a hadoop cluster, but we keep
>> getting the same status. The job passes the initial mahout stages
>> and
>> when it comes to be executed as a MR job, it seems to be stuck at
>> 0%
>> progress. Through the UI we see that it is submitted but not
>> running.
>> After a while it gets killed. In the logs the error shown is this one:
>>
>> 2011-09-21 07:47:50,507 INFO org.apache.hadoop.mapred.JobTracker:
>> problem cleaning system directory:
>> hdfs://master/var/lib/hadoop-0.20/cache/hdfs/mapred/system
>> org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot
>> create
>> directory /var/lib/hadoop-0.20/cache/hdfs/mapred/system. Name nod$
>> The reported blocks 0 needs additional 12 blocks to reach the
>> threshold
>> 0.9990 of total blocks 13. Safe mode will be turned off automatically.
>>          at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:1966)
>>          at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:1940)
>>          at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs(NameNode.java:770)
>>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>          at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>          at java.lang.reflect.Method.invoke(Method.java:597)
>>
>>
>> Some staging files seem to have been created however.
>>
>> I was thinking of sending this to the mahout mailing list but it
>> seems a
>> more core hadoop issue.
>>
>> We are using the following command to launch the mahout example:
>> ./mahout org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
>> --input hdfs://master/user/hdfs/testdata/synthetic_control.data --
>> output
>> hdfs://master/user/hdfs/testdata/output --t1 0.5 --t2 1 --maxIter 50
>>
>> Any clues?
>> George
>>
>> --
>>
>> ---------------------------
>>
>> George Kousiouris
>> Electrical and Computer Engineer
>> Division of Communications,
>> Electronics and Information Engineering
>> School of Electrical and Computer Engineering
>> Tel: +30 210 772 2546
>> Mobile: +30 6939354121
>> Fax: +30 210 772 2569
George Kousiouris
Electrical and Computer Engineer
Division of Communications,
Electronics and Information Engineering
School of Electrical and Computer Engineering
Tel: +30 210 772 2546
Mobile: +30 6939354121
Fax: +30 210 772 2569
Email: [EMAIL PROTECTED]
Site: http://users.ntua.gr/gkousiou/

National Technical University of Athens
9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB