Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - HDFS Quota


+
tatebet@... 2010-01-25, 09:01
Copy link to this message
-
Re: HDFS Quota
Boris Shkolnik 2010-01-25, 20:20
>>> /user/root/input is exceeded: namespace quota=-1 file count=2,
>From your example it looks like you have namespace quota (not diskspace),
i.e. quota for quantities of files (not size)
> Quota exceed exception,but it creates file of size 0.
Yes, I think this is by design. When HDFS creates a file, it doesn't know
how big the file will be. So diskspace quota is checked on writing to the
file (not on creation).

>Q3:Why did input file's capacity exceed the limit(space quota)?
What is your block size. I don't think it make sense to put quota for less
then a blocksize.

Boris.

On 1/25/10 1:01 AM, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:

> Hi everyone,
>
> I set Space Quotas for the amount of space on HDFS.
> But I have some questions.
>
> ■Question
> Q1:Why does diskspace become MByet when the file of KByte is input?(1024
> calculations in HDFS)
> Q2:Is there a person who has information for this problem that quota exceed
> exception,but it creates file of size 0?
> Q3:Why did input file's capacity exceed the limit(space quota)?
>   
>
> ■Details of question
> ★First time
> Even when I put file which was smaller than the capacity set in Space Quota,
> it became an error.
>
> The input file's capacity which I put is 64000=64kbyte.
>
> ・input file
> $ ls -ltr
> $ -rw-r--r-- 1 root root  64000 Jan 21 04:42 xaa
> $ du -h xaa
> 68K     xaa
>
> ・hdfs-site.xml(replication)
> <property>
>   <name>dfs.replication</name>
>   <value>2</value>
>  </property>
>
> ・Space Quota
> $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
> $ ./bin/hadoop fs -count -q /user/root/input/
>         none             inf          192000          192000            1
> 0                  0 hdfs://drbd-test-vm03/user/root/input
> $ ./bin/hadoop dfs -put input/xaa /user/root/input/
> 10/01/21 19:35:58 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.hdfs.protocol.QuotaExceededException:
> org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> quota=192000 diskspace=134217728
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccesso
> rImpl.java:39)
>         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructo
> rAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.jav
> a:96)
>         at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.ja
> va:58)
>         at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClien
> t.java:2875)
>         at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClie
> nt.java:2755)
>         at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:20
> 46)
>         at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.ja
> va:2232)
> Caused by: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> quota=192000 diskspace=134217728
>         at
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INo
> deDirectoryWithQuota.java:161)
>         at
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateNumItemsI
> nTree(INodeDirectoryWithQuota.java:134)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.jav
> a:859)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:2
> 65)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem
> .java:1427)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNames
+
tatebet@... 2010-01-26, 07:04
+
tatebet@... 2010-02-03, 09:20
+
tatebet@... 2010-02-19, 01:15
+
Eli Collins 2010-02-24, 00:24