Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Re: Re: HDFS block physical location


+
Chen He 2012-07-25, 22:20
Copy link to this message
-
Re: Re: HDFS block physical location
Hi JS,

You may also be interested in the following JIRA which proposes an API
for block->disk mapping information:
https://issues.apache.org/jira/browse/HDFS-3672

There has been some discussion about potential use cases on this JIRA.
If you can describe what your use case is for this information on the
JIRA, we'd really appreciate the input.

-Todd

On Wed, Jul 25, 2012 at 3:20 PM, Chen He <[EMAIL PROTECTED]> wrote:
> For block to filename mapping, you can get from my previous answer.
>
> For block to harddisk mapping, you may need to traverse all the directory
> that used for HDFS, I am sure your OS has the information about which hard
> drive is mounted to which directory.
>
> with these two types of information, you can write a small Perl or Python
> script to get what you want.
>
> Or
>
> Take look of the namenode.java and see where and how it saves the table of
> block information.
>
> Please correct me if there is any mistake.
>
> Chen
>
>
> On Wed, Jul 25, 2012 at 6:10 PM, <[EMAIL PROTECTED]> wrote:
>
>>
>> Thanks,
>> but that just gives me the hostnames or am I overlooking something?
>> I actually need the filename/harddisk on the node.
>>
>> JS
>>
>> Gesendet: Mittwoch, 25. Juli 2012 um 23:33 Uhr
>> Von: "Chen He" <[EMAIL PROTECTED]>
>> An: [EMAIL PROTECTED]
>> Betreff: Re: HDFS block physical location
>> >nohup hadoop fsck / -files -blocks -locations
>> >cat nohup.out | grep [your block name]
>>
>> Hope this helps.
>>
>> On Wed, Jul 25, 2012 at 5:17 PM, <[EMAIL PROTECTED]> wrote:
>>
>> > Hi,
>> >
>> > just a short question. Is there any way to figure out the physical
>> storage
>> > location of a given block?
>> > I don't mean just a list of hostnames (which I know how to obtain), but
>> > actually the file where it is being stored in.
>> > We use several hard disks for hdfs data on each node, and I would need to
>> > know which block ends up on which harddisk.
>> >
>> > Thanks!
>> > JS
>> >
>> >
>>
>>
>>
>>

--
Todd Lipcon
Software Engineer, Cloudera
+
20seconds@... 2012-07-25, 21:17
+
Chen He 2012-07-25, 21:33