Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> Feature request to provide DFSInputStream subclassing mechanism

Jeff Dost 2013-08-07, 17:59
Todd Lipcon 2013-08-07, 18:25
Joe Bounour 2013-08-07, 18:11
Andrew Wang 2013-08-07, 18:30
Jeff Dost 2013-08-07, 22:29
Copy link to this message
Re: Feature request to provide DFSInputStream subclassing mechanism
Blocks are supposed to be an internal abstraction within HDFS, and aren't
an inherent part of FileSystem (the user-visible class used to access all
Hadoop filesystems).

Is it possible to instead deal with files and offsets? On a read failure,
you could open a stream to the same file on the backup filesystem, seek to
the old file position, and retry the read. This feels like it's possible
via wrapping.
On Wed, Aug 7, 2013 at 3:29 PM, Jeff Dost <[EMAIL PROTECTED]> wrote:

> Thank you for the suggestion, but we don't see how simply wrapping a
> FileSystem object would be sufficient in our use case.  The reason why is
> we need to catch and handle read exceptions at the block level.  There
> aren't any public methods available in the high level FileSystem
> abstraction layer that would give us the fine grained control we need at
> block level read failures.
> Perhaps if I outline the steps more clearly it will help explain what we
> are trying to do.  Without our enhancements, suppose a user opens a file
> stream and starts reading the file from Hadoop. After some time, at some
> position into the file, if there happen to be no replicas available for a
> particular block for whatever reason, datanodes have gone down due to disk
> issues, etc. the stream will throw an IOException (BlockMissingException or
> similar) and the read will fail.
> What we are doing is rather than letting the stream fail, we have another
> stream queued up that knows how to fetch the blocks elsewhere outside of
> our Hadoop cluster that couldn't be retrieved.  So we need to be able to
> catch the exception at this point, and these externally fetched bytes then
> get read into the user supplied read buffer.  Now Hadoop can proceed to
> read in the stream the next blocks in the file.
> So as you can see this method of fail over on demand allows an input
> stream to keep reading data, without having to start it all over again if a
> failure occurs (assuming the remote bytes were successfully fetched).
> As a final note I would like to mention that we will be providing our
> failover module to the Open Science Grid.  Since we hope to provide this as
> a benefit to all OSG users running at participating T2 computing clusters,
> we will be committed to maintaining this software and any changes to Hadoop
> needed to make it work.  In other words we will be willing to maintain any
> implementation changes that may become necessary as Hadoop internals change
> in future releases.
> Thanks,
> Jeff
> On 8/7/13 11:30 AM, Andrew Wang wrote:
>> I don't think exposing DFSClient and DistributedFileSystem members is
>> necessary to achieve what you're trying to do. We've got wrapper
>> FileSystems like FilterFileSystem and ViewFileSystem which you might be
>> able to use for inspiration, and the HCFS wiki lists some third-party
>> FileSystems that might also be helpful too.
>> On Wed, Aug 7, 2013 at 11:11 AM, Joe Bounour <[EMAIL PROTECTED]> wrote:
>>  Hello Jeff
>>> Is it something that could go under HCFS project?
>>> http://wiki.apache.org/hadoop/**HCFS<http://wiki.apache.org/hadoop/HCFS>
>>> (I might be wrong?)
>>> Joe
>>> On 8/7/13 10:59 AM, "Jeff Dost" <[EMAIL PROTECTED]> wrote:
>>>  Hello,
>>>> We work in a software development team at the UCSD CMS Tier2 Center.  We
>>>> would like to propose a mechanism to allow one to subclass the
>>>> DFSInputStream in a clean way from an external package.  First I'd like
>>>> to give some motivation on why and then will proceed with the details.
>>>> We have a 3 Petabyte Hadoop cluster we maintain for the LHC experiment
>>>> at CERN.  There are other T2 centers worldwide that contain mirrors of
>>>> the same data we host.  We are working on an extension to Hadoop that,
>>>> on reading a file, if it is found that there are no available replicas
>>>> of a block, we use an external interface to retrieve this block of the
>>>> file from another data center.  The external interface is necessary
Matevz Tadel 2013-08-08, 18:52
Colin McCabe 2013-08-08, 19:10
Matevz Tadel 2013-08-08, 21:04
Suresh Srinivas 2013-08-08, 21:17
Steve Loughran 2013-08-08, 20:30
Matevz Tadel 2013-08-09, 04:51
Steve Loughran 2013-08-09, 17:31
Jeff Dost 2013-08-09, 19:52