Hi.

How can I get the free / used space on DFS, via Java?

What are the functions that can be used for that?

Note, I'm using a regular (non-super) user, so I need to do it in a similar
way to dfshealth.jsp, which AFAIK doesn't require any permissions.

Thanks in advance.

On Sun, Aug 23, 2009 at 7:22 AM, Stas Oskin<[EMAIL PROTECTED]> wrote:
> Hi.
>
> How can I get the free / used space on DFS, via Java?
>
> What are the functions that can be used for that?
>
> Note, I'm using a regular (non-super) user, so I need to do it in a similar
> way to dfshealth.jsp, which AFAIK doesn't require any permissions.
>
> Thanks in advance.
>

One way you can do this is thought JMX.

http://www.jointhegrid.com/svn/hadoop-cacti-jtg/trunk/src/com/jointhegrid/hadoopjmx/NameNodeStatistics.java

That is part of a cacti graphing application for hadoop I wrote
(http://www.jointhegrid.com/hadoop/)

Enjoy,

You can try something like this:
if (_FileSystem instanceof DistributedFileSystem)
        {
            DistributedFileSystem dfs = (DistributedFileSystem) _FileSystems;
            DiskStatus ds = dfs.getDiskStatus();
            long capacity = ds.getCapacity();
            long used = ds.getDfsUsed();
            long remaining = ds.getRemaining();
            long presentCapacity = used + remaining;

            hdfsPercentDiskUsed = Math.round((((1.0 * used) / presentCapacity) * 100));
        }

Arvind

________________________________
From: Stas Oskin <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Sunday, August 23, 2009 4:22:26 AM
Subject: Getting free space percentage on DFS

Hi.

How can I get the free / used space on DFS, via Java?

What are the functions that can be used for that?

Note, I'm using a regular (non-super) user, so I need to do it in a similar
way to dfshealth.jsp, which AFAIK doesn't require any permissions.

Thanks in advance.

      

Hi.

Thank you both for the advices - any idea if these approaches works for
non-super user?

Regards.

The APIs work for the user with which Hadoop was started. And moreover I don't think the User level authentication is there yet in Hadoop (not sure here though) for APIs...
________________________________
From: Stas Oskin <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Sunday, August 23, 2009 1:33:38 PM
Subject: Re: Getting free space percentage on DFS

Hi.

Thank you both for the advices - any idea if these approaches works for
non-super user?

Regards.

      

Hi.

2009/8/23 Arvind Sharma <[EMAIL PROTECTED]>

> You can try something like this:
>
>
> if (_FileSystem instanceof DistributedFileSystem)
>        {
>            DistributedFileSystem dfs = (DistributedFileSystem)
> _FileSystems;
>            DiskStatus ds = dfs.getDiskStatus();
>            long capacity = ds.getCapacity();
>            long used = ds.getDfsUsed();
>            long remaining = ds.getRemaining();
>            long presentCapacity = used + remaining;
>
>            hdfsPercentDiskUsed = Math.round((((1.0 * used) /
> presentCapacity) * 100));
>        }
>

 I just tested it, and I'm getting the exception that I'm not the superuser
- "Superuser privilege is required".

Any idea how to bypass this?

Thanks in advance!

Hi.

One way you can do this is thought JMX.
>
>
> http://www.jointhegrid.com/svn/hadoop-cacti-jtg/trunk/src/com/jointhegrid/hadoopjmx/NameNodeStatistics.java
>
> That is part of a cacti graphing application for hadoop I wrote
> (http://www.jointhegrid.com/hadoop/)
>
> Enjoy,
>

I'm going to try this next - you probably meant these links?

http://www.jointhegrid.com/svn/hadoop-cacti-jtg/trunk/src/com/jointhegrid/hadoopjmx/FSNamesystemStatus.java

http://www.jointhegrid.com/svn/hadoop-cacti-jtg/trunk/src/com/jointhegrid/hadoopjmx/FSDatasetStatus.java
(inherits
http://www.jointhegrid.com/svn/hadoop-cacti-jtg/trunk/src/com/jointhegrid/hadoopjmx/JMXBase.java
)

For JMX you can also look at JMXGet.java class. You can use this object to
get the data thru JMX.

Boris

On 8/24/09 1:27 AM, "Stas Oskin" <[EMAIL PROTECTED]> wrote:

> Hi.
>
> One way you can do this is thought JMX.
>>
>>
>> http://www.jointhegrid.com/svn/hadoop-cacti-jtg/trunk/src/com/jointhegrid/had
>> oopjmx/NameNodeStatistics.java
>>
>> That is part of a cacti graphing application for hadoop I wrote
>> (http://www.jointhegrid.com/hadoop/)
>>
>> Enjoy,
>>
>
> I'm going to try this next - you probably meant these links?
>
> http://www.jointhegrid.com/svn/hadoop-cacti-jtg/trunk/src/com/jointhegrid/hado
> opjmx/FSNamesystemStatus.java
>
> http://www.jointhegrid.com/svn/hadoop-cacti-jtg/trunk/src/com/jointhegrid/hado
> opjmx/FSDatasetStatus.java
> (inherits
> http://www.jointhegrid.com/svn/hadoop-cacti-jtg/trunk/src/com/jointhegrid/hado
> opjmx/JMXBase.java
> )