HDFS, mail # user - Re: Cant get HDFS to load more than 1Gig total data

Cant get HDFS to load more than 1Gig total data

Scott 2010-01-07, 16:34

Hi, new to hadoop/HDFS. We have a 4 node test cluster, and I can't
write more than approximately 1 Gig of total data. Any additional file
puts error out with the following:

WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/ads3x11-1256301562.log.lzo could only be replicated to 0
nodes, instead of 1

I have checked quotas and found none. I have also tried other users,
including the hadoop user, and get the same result. Any ideas?

Thanks,
Scott

Re: Cant get HDFS to load more than 1Gig total data

Allen Wittenauer 2010-01-07, 16:49

On 1/7/10 8:34 AM, "Scott" <[EMAIL PROTECTED]> wrote:
> WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/hadoop/ads3x11-1256301562.log.lzo could only be replicated to 0
> nodes, instead of 1

This almost always means your HDFS is in safemode and/or has no live
datanodes.

> I have checked quotas and found none. I have also tried other users,
> including the hadoop user, and get the same result. Any ideas?

How is the namenode heap?

Are you out of physical space?

What does hadoop fsck / say?

Re: Cant get HDFS to load more than 1Gig total data

Scott 2010-01-07, 21:11

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
  <title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Thanks for the reply Allen. I fixed it, though I am not sure of the
exact cause as I found more than one thing wrong. There were some
directories owned by root instead of hadoop that I think were the main
issue.<br>
<br>
Allen Wittenauer wrote:
<blockquote cite="mid:C76B50A1.223C%[EMAIL PROTECTED]"
type="cite">
  <pre wrap="">

On 1/7/10 8:34 AM, "Scott" <a class="moz-txt-link-rfc2396E" href="mailto:[EMAIL PROTECTED]"><[EMAIL PROTECTED]></a> wrote:
  </pre>
  <blockquote type="cite">
   <pre wrap=""> WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/ads3x11-1256301562.log.lzo could only be replicated to 0
nodes, instead of 1
   </pre>
  </blockquote>
  <pre wrap="">
This almost always means your HDFS is in safemode and/or has no live
datanodes.

  </pre>
  <blockquote type="cite">
   <pre wrap="">I have checked quotas and found none. I have also tried other users,
including the hadoop user, and get the same result. Any ideas?
   </pre>
  </blockquote>
  <pre wrap="">
How is the namenode heap?

Are you out of physical space?

What does hadoop fsck / say?

  </pre>
</blockquote>
</body>
</html>