Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> HDFS Quota


+
tatebet@... 2010-01-25, 09:01
>>> /user/root/input is exceeded: namespace quota=-1 file count=2,
>From your example it looks like you have namespace quota (not diskspace),
i.e. quota for quantities of files (not size)
> Quota exceed exception,but it creates file of size 0.
Yes, I think this is by design. When HDFS creates a file, it doesn't know
how big the file will be. So diskspace quota is checked on writing to the
file (not on creation).

>Q3:Why did input file's capacity exceed the limit(space quota)?
What is your block size. I don't think it make sense to put quota for less
then a blocksize.

Boris.

On 1/25/10 1:01 AM, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:

> Hi everyone,
>
> I set Space Quotas for the amount of space on HDFS.
> But I have some questions.
>
> ■Question
> Q1:Why does diskspace become MByet when the file of KByte is input?(1024
> calculations in HDFS)
> Q2:Is there a person who has information for this problem that quota exceed
> exception,but it creates file of size 0?
> Q3:Why did input file's capacity exceed the limit(space quota)?
>   
>
> ■Details of question
> ★First time
> Even when I put file which was smaller than the capacity set in Space Quota,
> it became an error.
>
> The input file's capacity which I put is 64000=64kbyte.
>
> ・input file
> $ ls -ltr
> $ -rw-r--r-- 1 root root  64000 Jan 21 04:42 xaa
> $ du -h xaa
> 68K     xaa
>
> ・hdfs-site.xml(replication)
> <property>
>   <name>dfs.replication</name>
>   <value>2</value>
>  </property>
>
> ・Space Quota
> $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
> $ ./bin/hadoop fs -count -q /user/root/input/
>         none             inf          192000          192000            1
> 0                  0 hdfs://drbd-test-vm03/user/root/input
> $ ./bin/hadoop dfs -put input/xaa /user/root/input/
> 10/01/21 19:35:58 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.hdfs.protocol.QuotaExceededException:
> org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> quota=192000 diskspace=134217728
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccesso
> rImpl.java:39)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructo
> rAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.jav
> a:96)
>         at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.ja
> va:58)
>         at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClien
> t.java:2875)
>         at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClie
> nt.java:2755)
>         at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:20
> 46)
>         at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.ja
> va:2232)
> Caused by: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> quota=192000 diskspace=134217728
>         at
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INo
> deDirectoryWithQuota.java:161)
>         at
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateNumItemsI
> nTree(INodeDirectoryWithQuota.java:134)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.jav
> a:859)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:2
> 65)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem
> .java:1427)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNames
+
tatebet@... 2010-01-26, 07:04
+
tatebet@... 2010-02-03, 09:20
+
tatebet@... 2010-02-19, 01:15
+
Eli Collins 2010-02-24, 00:24
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB