MapReduce, mail # user - sym Links in hadoop

Re: sym Links in hadoop

Dave Beech 2012-10-08, 13:30

Hi,
The FileUtil.symlink command does nothing more than call the unix "ln"
command, so it has no knowledge of how to work with Hadoop archive
files, only plain files and directories.
Is your archive on local disk, or in HDFS?
Cheers,
Dave

On 8 October 2012 13:43, Visioner Sadak <[EMAIL PROTECTED]> wrote:
> I tried using FileUtil class for creating a symlink within hadoop actually i
> want to create a symlink for my har directory so my code looks like
>
> FileUtil.symLink("/user2/","har:///user/5oct2012.har")
>
> but getting error like this
>
> org.apache.hadoop.fs.FileUtil
> - Command 'ln -s /user2/ har://user/5oct2012.har/' failed 1 with: ln:
> target `har:///user/5oct2012.har/' is not a directory: No such file or
> directory
>
>
>
>
> On Mon, Oct 8, 2012 at 12:13 PM, Visioner Sadak <[EMAIL PROTECTED]>
> wrote:
>>
>> Hello,
>>
>> can we create symlinks within hadoop is ther any shell commands
>> or can we do it thru java....
>
>

Re: sym Links in hadoop

Visioner Sadak 2012-10-08, 14:20

thanks dave its in hdfs only....any other methods of creating a symlink

On Mon, Oct 8, 2012 at 7:00 PM, Dave Beech <[EMAIL PROTECTED]> wrote:

> Hi,
> The FileUtil.symlink command does nothing more than call the unix "ln"
> command, so it has no knowledge of how to work with Hadoop archive
> files, only plain files and directories.
> Is your archive on local disk, or in HDFS?
> Cheers,
> Dave
>
> On 8 October 2012 13:43, Visioner Sadak <[EMAIL PROTECTED]> wrote:
> > I tried using FileUtil class for creating a symlink within hadoop
> actually i
> > want to create a symlink for my har directory so my code looks like
> >
> > FileUtil.symLink("/user2/","har:///user/5oct2012.har")
> >
> > but getting error like this
> >
> > org.apache.hadoop.fs.FileUtil
> > - Command 'ln -s /user2/ har://user/5oct2012.har/' failed 1 with: ln:
> > target `har:///user/5oct2012.har/' is not a directory: No such file or
> > directory
> >
> >
> >
> >
> > On Mon, Oct 8, 2012 at 12:13 PM, Visioner Sadak <
> [EMAIL PROTECTED]>
> > wrote:
> >>
> >> Hello,
> >>
> >> can we create symlinks within hadoop is ther any shell
> commands
> >> or can we do it thru java....
> >
> >
>

Re: sym Links in hadoop

Terry Healy 2012-10-08, 15:33

Visioner-

I hope it is not heresy to mention here, but I believe the MapR
implementation of Hadoop supports symlink via NFS.

-Terry

On 10/08/2012 10:20 AM, Visioner Sadak wrote:
> thanks dave its in hdfs only....any other methods of creating a symlink
>
> On Mon, Oct 8, 2012 at 7:00 PM, Dave Beech <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
> Hi,
> The FileUtil.symlink command does nothing more than call the unix "ln"
> command, so it has no knowledge of how to work with Hadoop archive
> files, only plain files and directories.
> Is your archive on local disk, or in HDFS?
> Cheers,
> Dave
>
> On 8 October 2012 13:43, Visioner Sadak <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
> > I tried using FileUtil class for creating a symlink within hadoop
> actually i
> > want to create a symlink for my har directory so my code looks like
> >
> > FileUtil.symLink("/user2/","har:///user/5oct2012.har")
> >
> > but getting error like this
> >
> > org.apache.hadoop.fs.FileUtil
> > - Command 'ln -s /user2/ har://user/5oct2012.har/' failed 1 with: ln:
> > target `har:///user/5oct2012.har/' is not a directory: No such file or
> > directory
> >
> >
> >
> >
> > On Mon, Oct 8, 2012 at 12:13 PM, Visioner Sadak
> <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
> > wrote:
> >>
> >> Hello,
> >>
> >> can we create symlinks within hadoop is ther any shell
> commands
> >> or can we do it thru java....
> >
> >
>
>

--
Terry Healy / [EMAIL PROTECTED]
Cyber Security Operations
Brookhaven National Laboratory
Building 515, Upton N.Y. 11973

Re: sym Links in hadoop

Colin McCabe 2012-10-08, 18:32

You can create an HDFS symlink by using the FileContext#createSymlink
function. I don't think this can be done through the "hadoop fs"
command, so you're going to have to write some Java code to do this.

We should consider adding this functionality to the "hadoop fs"
command in the future.

Colin
On Mon, Oct 8, 2012 at 7:20 AM, Visioner Sadak <[EMAIL PROTECTED]> wrote:
> thanks dave its in hdfs only....any other methods of creating a symlink
>
>
> On Mon, Oct 8, 2012 at 7:00 PM, Dave Beech <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>> The FileUtil.symlink command does nothing more than call the unix "ln"
>> command, so it has no knowledge of how to work with Hadoop archive
>> files, only plain files and directories.
>> Is your archive on local disk, or in HDFS?
>> Cheers,
>> Dave
>>
>> On 8 October 2012 13:43, Visioner Sadak <[EMAIL PROTECTED]> wrote:
>> > I tried using FileUtil class for creating a symlink within hadoop
>> > actually i
>> > want to create a symlink for my har directory so my code looks like
>> >
>> > FileUtil.symLink("/user2/","har:///user/5oct2012.har")
>> >
>> > but getting error like this
>> >
>> > org.apache.hadoop.fs.FileUtil
>> > - Command 'ln -s /user2/ har://user/5oct2012.har/' failed 1 with: ln:
>> > target `har:///user/5oct2012.har/' is not a directory: No such file or
>> > directory
>> >
>> >
>> >
>> >
>> > On Mon, Oct 8, 2012 at 12:13 PM, Visioner Sadak
>> > <[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> can we create symlinks within hadoop is ther any shell
>> >> commands
>> >> or can we do it thru java....
>> >
>> >
>
>

Re: sym Links in hadoop

Visioner Sadak 2012-10-09, 06:52

Thanks Colin I tried using FileContext but the class is showing as
depricated

On Tue, Oct 9, 2012 at 12:02 AM, Colin McCabe <[EMAIL PROTECTED]>wrote:

> You can create an HDFS symlink by using the FileContext#createSymlink
> function. I don't think this can be done through the "hadoop fs"
> command, so you're going to have to write some Java code to do this.
>
> We should consider adding this functionality to the "hadoop fs"
> command in the future.
>
> Colin
>
>
> On Mon, Oct 8, 2012 at 7:20 AM, Visioner Sadak <[EMAIL PROTECTED]>
> wrote:
> > thanks dave its in hdfs only....any other methods of creating a symlink
> >
> >
> > On Mon, Oct 8, 2012 at 7:00 PM, Dave Beech <[EMAIL PROTECTED]> wrote:
> >>
> >> Hi,
> >> The FileUtil.symlink command does nothing more than call the unix "ln"
> >> command, so it has no knowledge of how to work with Hadoop archive
> >> files, only plain files and directories.
> >> Is your archive on local disk, or in HDFS?
> >> Cheers,
> >> Dave
> >>
> >> On 8 October 2012 13:43, Visioner Sadak <[EMAIL PROTECTED]>
> wrote:
> >> > I tried using FileUtil class for creating a symlink within hadoop
> >> > actually i
> >> > want to create a symlink for my har directory so my code looks like
> >> >
> >> > FileUtil.symLink("/user2/","har:///user/5oct2012.har")
> >> >
> >> > but getting error like this
> >> >
> >> > org.apache.hadoop.fs.FileUtil
> >> > - Command 'ln -s /user2/ har://user/5oct2012.har/' failed 1 with: ln:
> >> > target `har:///user/5oct2012.har/' is not a directory: No such file or
> >> > directory
> >> >
> >> >
> >> >
> >> >
> >> > On Mon, Oct 8, 2012 at 12:13 PM, Visioner Sadak
> >> > <[EMAIL PROTECTED]>
> >> > wrote:
> >> >>
> >> >> Hello,
> >> >>
> >> >> can we create symlinks within hadoop is ther any shell
> >> >> commands
> >> >> or can we do it thru java....
> >> >
> >> >
> >
> >
>

Re: sym Links in hadoop

Visioner Sadak 2012-10-09, 08:28

I am using hadoop 1.0.3

On Tue, Oct 9, 2012 at 12:22 PM, Visioner Sadak <[EMAIL PROTECTED]>wrote:

> Thanks Colin I tried using FileContext but the class is showing as
> depricated
>
> On Tue, Oct 9, 2012 at 12:02 AM, Colin McCabe <[EMAIL PROTECTED]>wrote:
>
>> You can create an HDFS symlink by using the FileContext#createSymlink
>> function. I don't think this can be done through the "hadoop fs"
>> command, so you're going to have to write some Java code to do this.
>>
>> We should consider adding this functionality to the "hadoop fs"
>> command in the future.
>>
>> Colin
>>
>>
>> On Mon, Oct 8, 2012 at 7:20 AM, Visioner Sadak <[EMAIL PROTECTED]>
>> wrote:
>> > thanks dave its in hdfs only....any other methods of creating a symlink
>> >
>> >
>> > On Mon, Oct 8, 2012 at 7:00 PM, Dave Beech <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hi,
>> >> The FileUtil.symlink command does nothing more than call the unix "ln"
>> >> command, so it has no knowledge of how to work with Hadoop archive
>> >> files, only plain files and directories.
>> >> Is your archive on local disk, or in HDFS?
>> >> Cheers,
>> >> Dave
>> >>
>> >> On 8 October 2012 13:43, Visioner Sadak <[EMAIL PROTECTED]>
>> wrote:
>> >> > I tried using FileUtil class for creating a symlink within hadoop
>> >> > actually i
>> >> > want to create a symlink for my har directory so my code looks like
>> >> >
>> >> > FileUtil.symLink("/user2/","har:///user/5oct2012.har")
>> >> >
>> >> > but getting error like this
>> >> >
>> >> > org.apache.hadoop.fs.FileUtil
>> >> > - Command 'ln -s /user2/ har://user/5oct2012.har/' failed 1 with:
>> ln:
>> >> > target `har:///user/5oct2012.har/' is not a directory: No such file
>> or
>> >> > directory
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Oct 8, 2012 at 12:13 PM, Visioner Sadak
>> >> > <[EMAIL PROTECTED]>
>> >> > wrote:
>> >> >>
>> >> >> Hello,
>> >> >>
>> >> >> can we create symlinks within hadoop is ther any shell
>> >> >> commands
>> >> >> or can we do it thru java....
>> >> >
>> >> >
>> >
>> >
>>
>
>

Re: sym Links in hadoop

Dave Beech 2012-10-09, 12:40

A lot of code in Hadoop is marked "Deprecated". This doesn't mean you
shouldn't use it ;)

Cheers,
Dave

On 9 October 2012 09:28, Visioner Sadak <[EMAIL PROTECTED]> wrote:
> I am using hadoop 1.0.3
>
> On Tue, Oct 9, 2012 at 12:22 PM, Visioner Sadak <[EMAIL PROTECTED]>
> wrote:
>>
>> Thanks Colin I tried using FileContext but the class is showing as
>> depricated
>>
>> On Tue, Oct 9, 2012 at 12:02 AM, Colin McCabe <[EMAIL PROTECTED]>
>> wrote:
>>>
>>> You can create an HDFS symlink by using the FileContext#createSymlink
>>> function. I don't think this can be done through the "hadoop fs"
>>> command, so you're going to have to write some Java code to do this.
>>>
>>> We should consider adding this functionality to the "hadoop fs"
>>> command in the future.
>>>
>>> Colin
>>>
>>>
>>> On Mon, Oct 8, 2012 at 7:20 AM, Visioner Sadak <[EMAIL PROTECTED]>
>>> wrote:
>>> > thanks dave its in hdfs only....any other methods of creating a symlink
>>> >
>>> >
>>> > On Mon, Oct 8, 2012 at 7:00 PM, Dave Beech <[EMAIL PROTECTED]> wrote:
>>> >>
>>> >> Hi,
>>> >> The FileUtil.symlink command does nothing more than call the unix "ln"
>>> >> command, so it has no knowledge of how to work with Hadoop archive
>>> >> files, only plain files and directories.
>>> >> Is your archive on local disk, or in HDFS?
>>> >> Cheers,
>>> >> Dave
>>> >>
>>> >> On 8 October 2012 13:43, Visioner Sadak <[EMAIL PROTECTED]>
>>> >> wrote:
>>> >> > I tried using FileUtil class for creating a symlink within hadoop
>>> >> > actually i
>>> >> > want to create a symlink for my har directory so my code looks like
>>> >> >
>>> >> > FileUtil.symLink("/user2/","har:///user/5oct2012.har")
>>> >> >
>>> >> > but getting error like this
>>> >> >
>>> >> > org.apache.hadoop.fs.FileUtil
>>> >> > - Command 'ln -s /user2/ har://user/5oct2012.har/' failed 1 with:
>>> >> > ln:
>>> >> > target `har:///user/5oct2012.har/' is not a directory: No such file
>>> >> > or
>>> >> > directory
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Mon, Oct 8, 2012 at 12:13 PM, Visioner Sadak
>>> >> > <[EMAIL PROTECTED]>
>>> >> > wrote:
>>> >> >>
>>> >> >> Hello,
>>> >> >>
>>> >> >> can we create symlinks within hadoop is ther any shell
>>> >> >> commands
>>> >> >> or can we do it thru java....
>>> >> >
>>> >> >
>>> >
>>> >
>>
>>
>

Re: sym Links in hadoop

Dave Beech 2012-10-09, 12:44

Actually, I don't think the FileContext class Colin mentioned is the
one you are talking about.

Hadoop 1.0.3 doesn't have org.apache.hadoop.fs.FileContext, which is
the class you would need, but it does have
org.apache.hadoop.metrics.file.FileContext which is something
completely different.

Symlink support for HDFS, as far as I'm aware, isn't contained in any
"stable" release of Hadoop (i.e. 1.x.x)

On 9 October 2012 13:40, Dave Beech <[EMAIL PROTECTED]> wrote:
> A lot of code in Hadoop is marked "Deprecated". This doesn't mean you
> shouldn't use it ;)
>
> Cheers,
> Dave
>
> On 9 October 2012 09:28, Visioner Sadak <[EMAIL PROTECTED]> wrote:
>> I am using hadoop 1.0.3
>>
>> On Tue, Oct 9, 2012 at 12:22 PM, Visioner Sadak <[EMAIL PROTECTED]>
>> wrote:
>>>
>>> Thanks Colin I tried using FileContext but the class is showing as
>>> depricated
>>>
>>> On Tue, Oct 9, 2012 at 12:02 AM, Colin McCabe <[EMAIL PROTECTED]>
>>> wrote:
>>>>
>>>> You can create an HDFS symlink by using the FileContext#createSymlink
>>>> function. I don't think this can be done through the "hadoop fs"
>>>> command, so you're going to have to write some Java code to do this.
>>>>
>>>> We should consider adding this functionality to the "hadoop fs"
>>>> command in the future.
>>>>
>>>> Colin
>>>>
>>>>
>>>> On Mon, Oct 8, 2012 at 7:20 AM, Visioner Sadak <[EMAIL PROTECTED]>
>>>> wrote:
>>>> > thanks dave its in hdfs only....any other methods of creating a symlink
>>>> >
>>>> >
>>>> > On Mon, Oct 8, 2012 at 7:00 PM, Dave Beech <[EMAIL PROTECTED]> wrote:
>>>> >>
>>>> >> Hi,
>>>> >> The FileUtil.symlink command does nothing more than call the unix "ln"
>>>> >> command, so it has no knowledge of how to work with Hadoop archive
>>>> >> files, only plain files and directories.
>>>> >> Is your archive on local disk, or in HDFS?
>>>> >> Cheers,
>>>> >> Dave
>>>> >>
>>>> >> On 8 October 2012 13:43, Visioner Sadak <[EMAIL PROTECTED]>
>>>> >> wrote:
>>>> >> > I tried using FileUtil class for creating a symlink within hadoop
>>>> >> > actually i
>>>> >> > want to create a symlink for my har directory so my code looks like
>>>> >> >
>>>> >> > FileUtil.symLink("/user2/","har:///user/5oct2012.har")
>>>> >> >
>>>> >> > but getting error like this
>>>> >> >
>>>> >> > org.apache.hadoop.fs.FileUtil
>>>> >> > - Command 'ln -s /user2/ har://user/5oct2012.har/' failed 1 with:
>>>> >> > ln:
>>>> >> > target `har:///user/5oct2012.har/' is not a directory: No such file
>>>> >> > or
>>>> >> > directory
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Mon, Oct 8, 2012 at 12:13 PM, Visioner Sadak
>>>> >> > <[EMAIL PROTECTED]>
>>>> >> > wrote:
>>>> >> >>
>>>> >> >> Hello,
>>>> >> >>
>>>> >> >> can we create symlinks within hadoop is ther any shell
>>>> >> >> commands
>>>> >> >> or can we do it thru java....
>>>> >> >
>>>> >> >
>>>> >
>>>> >
>>>
>>>
>>

Re: sym Links in hadoop

Visioner Sadak 2012-10-09, 13:23

yes dave u r right man i think symlink support is not available for 1.x.x
rel ... i did searched a lot :) i cant move to lower versions
act i wanted webhdfs thts why i needed 1.x.x versions its a complete
deadlock for me know i guess coz i needed webhdfs plus symlink :) ....

On Tue, Oct 9, 2012 at 6:14 PM, Dave Beech <[EMAIL PROTECTED]> wrote:

> Actually, I don't think the FileContext class Colin mentioned is the
> one you are talking about.
>
> Hadoop 1.0.3 doesn't have org.apache.hadoop.fs.FileContext, which is
> the class you would need, but it does have
> org.apache.hadoop.metrics.file.FileContext which is something
> completely different.
>
> Symlink support for HDFS, as far as I'm aware, isn't contained in any
> "stable" release of Hadoop (i.e. 1.x.x)
>
> On 9 October 2012 13:40, Dave Beech <[EMAIL PROTECTED]> wrote:
> > A lot of code in Hadoop is marked "Deprecated". This doesn't mean you
> > shouldn't use it ;)
> >
> > Cheers,
> > Dave
> >
> > On 9 October 2012 09:28, Visioner Sadak <[EMAIL PROTECTED]>
> wrote:
> >> I am using hadoop 1.0.3
> >>
> >> On Tue, Oct 9, 2012 at 12:22 PM, Visioner Sadak <
> [EMAIL PROTECTED]>
> >> wrote:
> >>>
> >>> Thanks Colin I tried using FileContext but the class is showing as
> >>> depricated
> >>>
> >>> On Tue, Oct 9, 2012 at 12:02 AM, Colin McCabe <[EMAIL PROTECTED]>
> >>> wrote:
> >>>>
> >>>> You can create an HDFS symlink by using the FileContext#createSymlink
> >>>> function. I don't think this can be done through the "hadoop fs"
> >>>> command, so you're going to have to write some Java code to do this.
> >>>>
> >>>> We should consider adding this functionality to the "hadoop fs"
> >>>> command in the future.
> >>>>
> >>>> Colin
> >>>>
> >>>>
> >>>> On Mon, Oct 8, 2012 at 7:20 AM, Visioner Sadak <
> [EMAIL PROTECTED]>
> >>>> wrote:
> >>>> > thanks dave its in hdfs only....any other methods of creating a
> symlink
> >>>> >
> >>>> >
> >>>> > On Mon, Oct 8, 2012 at 7:00 PM, Dave Beech <[EMAIL PROTECTED]>
> wrote:
> >>>> >>
> >>>> >> Hi,
> >>>> >> The FileUtil.symlink command does nothing more than call the unix
> "ln"
> >>>> >> command, so it has no knowledge of how to work with Hadoop archive
> >>>> >> files, only plain files and directories.
> >>>> >> Is your archive on local disk, or in HDFS?
> >>>> >> Cheers,
> >>>> >> Dave
> >>>> >>
> >>>> >> On 8 October 2012 13:43, Visioner Sadak <[EMAIL PROTECTED]>
> >>>> >> wrote:
> >>>> >> > I tried using FileUtil class for creating a symlink within hadoop
> >>>> >> > actually i
> >>>> >> > want to create a symlink for my har directory so my code looks
> like
> >>>> >> >
> >>>> >> > FileUtil.symLink("/user2/","har:///user/5oct2012.har")
> >>>> >> >
> >>>> >> > but getting error like this
> >>>> >> >
> >>>> >> > org.apache.hadoop.fs.FileUtil
> >>>> >> > - Command 'ln -s /user2/ har://user/5oct2012.har/' failed 1
> with:
> >>>> >> > ln:
> >>>> >> > target `har:///user/5oct2012.har/' is not a directory: No such
> file
> >>>> >> > or
> >>>> >> > directory
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> >
> >>>> >> > On Mon, Oct 8, 2012 at 12:13 PM, Visioner Sadak
> >>>> >> > <[EMAIL PROTECTED]>
> >>>> >> > wrote:
> >>>> >> >>
> >>>> >> >> Hello,
> >>>> >> >>
> >>>> >> >> can we create symlinks within hadoop is ther any shell
> >>>> >> >> commands
> >>>> >> >> or can we do it thru java....
> >>>> >> >
> >>>> >> >
> >>>> >
> >>>> >
> >>>
> >>>
> >>
>