Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - symlink support in Hadoop 2 GA


Copy link to this message
-
Re: symlink support in Hadoop 2 GA
Colin McCabe 2013-09-17, 15:14
I think it makes sense to finish symlinks support in the Hadoop 2 GA release.

Colin

On Mon, Sep 16, 2013 at 6:49 PM, Andrew Wang <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> I wanted to broadcast plans for putting the FileSystem symlinks work
> (HADOOP-8040) into branch-2.1 for the pending Hadoop 2 GA release. I think
> it's pretty important we get it in since it's not a compatible change; if
> it misses the GA train, we're not going to have symlinks until the next
> major release.
>
> However, we're still dealing with ongoing issues revealed via testing.
> There's user-code out there that only handles files and directories and
> will barf when given a symlink (perhaps a dangling one!). See HADOOP-9912
> for a nice example where globStatus returning symlinks broke Pig; some of
> us had a conference call to talk it through, and one definite conclusion
> was that this wasn't solvable in a generally compatible manner.
>
> There are also still some gaps in symlink support right now. For example,
> the more esoteric FileSystems like WebHDFS, HttpFS, and HFTP need symlink
> resolution, and tooling like the FsShell and Distcp still need to be
> updated as well.
>
> So, there's definitely work to be done, but there are a lot of users
> interested in the feature, and symlinks really should be in GA. Would
> appreciate any thoughts/input on the matter.
>
> Thanks,
> Andrew