Hello,

I am working on a project involving hdfs and fuse-dfs API on top of it. I
wanted to trace through the functions called from libhdfs API by fuse-dfs
functions. I added print statements inside the hdfs.c file in appropriate
places to see how the functions progress. I execute ant compile-c++-libhdfs
-Dlibhdfs=1 and then ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
-Djava5.home=/usr/lib/jvm/java-1.5.0-sun. However, when I use fuse-dfs I
cannot see any of the print statements executed from libhdfs/hdfs.c.

I am using hadoop-0.20.2 version and the libhdfs is present in
hadoop-0.20.2/src/c++/libhdfs. Could someone tell me if this libhdfs is the
one compiled and used or if there will be some other libhdfs that is
accessed. If this is the one, then why are the changes made in its files
reflected on running the code?

Thanks,
Aastha.

--
Aastha Mehta
Intern, NetApp, Bangalore
4th year undergraduate, BITS Pilani
E-mail: [EMAIL PROTECTED]

Hi Aastha,

Try using "ldd" against the fuse_dfs executable, and see where you are pulling libhdfs.so from.  It may be it is linking from the "wrong one".

Brian

On Mar 17, 2011, at 3:24 PM, Aastha Mehta wrote:

> Hello,
>
> I am working on a project involving hdfs and fuse-dfs API on top of it. I
> wanted to trace through the functions called from libhdfs API by fuse-dfs
> functions. I added print statements inside the hdfs.c file in appropriate
> places to see how the functions progress. I execute ant compile-c++-libhdfs
> -Dlibhdfs=1 and then ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
> -Djava5.home=/usr/lib/jvm/java-1.5.0-sun. However, when I use fuse-dfs I
> cannot see any of the print statements executed from libhdfs/hdfs.c.
>
> I am using hadoop-0.20.2 version and the libhdfs is present in
> hadoop-0.20.2/src/c++/libhdfs. Could someone tell me if this libhdfs is the
> one compiled and used or if there will be some other libhdfs that is
> accessed. If this is the one, then why are the changes made in its files
> reflected on running the code?
>
> Thanks,
> Aastha.
>
> --
> Aastha Mehta
> Intern, NetApp, Bangalore
> 4th year undergraduate, BITS Pilani
> E-mail: [EMAIL PROTECTED]

Hi Brian,

Thanks for your reply. I checked the libhdfs.so to which it links. Following
is the output of the ldd command:

$ ldd ./fuse_dfs

linux-gate.so.1 =>  (0x009c3000)
libhdfs.so.0 => /home/hadoop/hadoop/hadoop-0.20.2/build/libhdfs/libhdfs.so.0
(0x00457000)
libfuse.so.2 => /usr/local/lib/libfuse.so.2 (0x00567000)
libjvm.so => /usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/client/libjvm.so
(0x00ef3000)
libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x00110000)
libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0x00c78000)
libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0x006ab000)
libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0x008d3000)
librt.so.1 => /lib/tls/i686/cmov/librt.so.1 (0x00ce8000)
/lib/ld-linux.so.2 (0x00ed6000)

Although the fuse_dfs_wrapper.sh script has specified the LD_LIBRARY_PATH
there, I had to do an explicit export of the variable so that libhdfs.so.0
and libjvm.so files above had links to the correct object files.

Could you please tell what could the problem be now?

Thanks,

Aastha.

On 18 March 2011 03:08, Brian Bockelman <[EMAIL PROTECTED]> wrote:

> Hi Aastha,
>
> Try using "ldd" against the fuse_dfs executable, and see where you are
> pulling libhdfs.so from.  It may be it is linking from the "wrong one".
>
> Brian
>
> On Mar 17, 2011, at 3:24 PM, Aastha Mehta wrote:
>
> > Hello,
> >
> > I am working on a project involving hdfs and fuse-dfs API on top of it. I
> > wanted to trace through the functions called from libhdfs API by fuse-dfs
> > functions. I added print statements inside the hdfs.c file in appropriate
> > places to see how the functions progress. I execute ant
> compile-c++-libhdfs
> > -Dlibhdfs=1 and then ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
> > -Djava5.home=/usr/lib/jvm/java-1.5.0-sun. However, when I use fuse-dfs I
> > cannot see any of the print statements executed from libhdfs/hdfs.c.
> >
> > I am using hadoop-0.20.2 version and the libhdfs is present in
> > hadoop-0.20.2/src/c++/libhdfs. Could someone tell me if this libhdfs is
> the
> > one compiled and used or if there will be some other libhdfs that is
> > accessed. If this is the one, then why are the changes made in its files
> > reflected on running the code?
> >
> > Thanks,
> > Aastha.
> >
> > --
> > Aastha Mehta
> > Intern, NetApp, Bangalore
> > 4th year undergraduate, BITS Pilani
> > E-mail: [EMAIL PROTECTED]
>
>
--
Aastha Mehta
Intern, NetApp, Bangalore
4th year undergraduate, BITS Pilani
E-mail: [EMAIL PROTECTED]

Hi Aastha,

This is not hadoop-specific advice, but general debugging:

1) Verify you are using the version of the library you think you are.  Done.
2) Rebuild.
3) Check the timestamp on the rebuilt source and verify it's from approximately now.

In trunk, the directory you point to is *not* where libhdfs is built.  It is here:

build/c++/lib/libhdfs.so

Don't know where you got your sources from; that might be an issue.

Brian

On Mar 18, 2011, at 4:58 AM, Aastha Mehta wrote:

> Hi Brian,
>
> Thanks for your reply. I checked the libhdfs.so to which it links. Following
> is the output of the ldd command:
>
> $ ldd ./fuse_dfs
>
> linux-gate.so.1 =>  (0x009c3000)
> libhdfs.so.0 => /home/hadoop/hadoop/hadoop-0.20.2/build/libhdfs/libhdfs.so.0
> (0x00457000)
> libfuse.so.2 => /usr/local/lib/libfuse.so.2 (0x00567000)
> libjvm.so => /usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/client/libjvm.so
> (0x00ef3000)
> libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x00110000)
> libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0x00c78000)
> libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0x006ab000)
> libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0x008d3000)
> librt.so.1 => /lib/tls/i686/cmov/librt.so.1 (0x00ce8000)
> /lib/ld-linux.so.2 (0x00ed6000)
>
> Although the fuse_dfs_wrapper.sh script has specified the LD_LIBRARY_PATH
> there, I had to do an explicit export of the variable so that libhdfs.so.0
> and libjvm.so files above had links to the correct object files.
>
> Could you please tell what could the problem be now?
>
> Thanks,
>
> Aastha.
>
> On 18 March 2011 03:08, Brian Bockelman <[EMAIL PROTECTED]> wrote:
>
>> Hi Aastha,
>>
>> Try using "ldd" against the fuse_dfs executable, and see where you are
>> pulling libhdfs.so from.  It may be it is linking from the "wrong one".
>>
>> Brian
>>
>> On Mar 17, 2011, at 3:24 PM, Aastha Mehta wrote:
>>
>>> Hello,
>>>
>>> I am working on a project involving hdfs and fuse-dfs API on top of it. I
>>> wanted to trace through the functions called from libhdfs API by fuse-dfs
>>> functions. I added print statements inside the hdfs.c file in appropriate
>>> places to see how the functions progress. I execute ant
>> compile-c++-libhdfs
>>> -Dlibhdfs=1 and then ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
>>> -Djava5.home=/usr/lib/jvm/java-1.5.0-sun. However, when I use fuse-dfs I
>>> cannot see any of the print statements executed from libhdfs/hdfs.c.
>>>
>>> I am using hadoop-0.20.2 version and the libhdfs is present in
>>> hadoop-0.20.2/src/c++/libhdfs. Could someone tell me if this libhdfs is
>> the
>>> one compiled and used or if there will be some other libhdfs that is
>>> accessed. If this is the one, then why are the changes made in its files
>>> reflected on running the code?
>>>
>>> Thanks,
>>> Aastha.
>>>
>>> --
>>> Aastha Mehta
>>> Intern, NetApp, Bangalore
>>> 4th year undergraduate, BITS Pilani
>>> E-mail: [EMAIL PROTECTED]
>>
>>
>
>
> --
> Aastha Mehta
> Intern, NetApp, Bangalore
> 4th year undergraduate, BITS Pilani
> E-mail: [EMAIL PROTECTED]

Hi Brian,

Thanks for your reply. I downloaded hadoop-0.20.2 from apache.org site only.
In my package the libhdfs is not present in the path you mentioned. It is
present in hadoop-0.20.2/c++/Linux-i386-32/lib/libhdfs.so. Further, I
created a directory libhdfs inside build directory manually and created
symlinks to libhdfs.so and libhdfs.so.0 from the above directory. This is
what i did:

mkdir $HADOOP_HOME/build/libhdfs
cd $HADOOP_HOME/build/libhdfs
ln -s $HADOOP_HOME/c++/Linux-i386-32/lib/libhdfs.so libhdfs.so
ln -s $HADOOP_HOME/c++/Linux-i386-32/lib/libhdfs.so.0 libhdfs.so.0

Could you tell me if something's wrong with this?

I also tried manually creating the directory you mentioned and put the
libhdfs files there. That doesn't work. Could you suggest to me what I
should do now? If I need to change the package I am using, please tell me
the link to the version which works.

Thanks again for your time.

Regards,

Aastha.

On 18 March 2011 18:53, Brian Bockelman <[EMAIL PROTECTED]> wrote:

> Hi Aastha,
>
> This is not hadoop-specific advice, but general debugging:
>
> 1) Verify you are using the version of the library you think you are.
>  Done.
> 2) Rebuild.
> 3) Check the timestamp on the rebuilt source and verify it's from
> approximately now.
>
> In trunk, the directory you point to is *not* where libhdfs is built.  It
> is here:
>
> build/c++/lib/libhdfs.so
>
> Don't know where you got your sources from; that might be an issue.
>
> Brian
>
> On Mar 18, 2011, at 4:58 AM, Aastha Mehta wrote:
>
> > Hi Brian,
> >
> > Thanks for your reply. I checked the libhdfs.so to which it links.
> Following
> > is the output of the ldd command:
> >
> > $ ldd ./fuse_dfs
> >
> > linux-gate.so.1 =>  (0x009c3000)
> > libhdfs.so.0 =>
> /home/hadoop/hadoop/hadoop-0.20.2/build/libhdfs/libhdfs.so.0
> > (0x00457000)
> > libfuse.so.2 => /usr/local/lib/libfuse.so.2 (0x00567000)
> > libjvm.so =>
> /usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/i386/client/libjvm.so
> > (0x00ef3000)
> > libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x00110000)
> > libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0x00c78000)
> > libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0x006ab000)
> > libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0x008d3000)
> > librt.so.1 => /lib/tls/i686/cmov/librt.so.1 (0x00ce8000)
> > /lib/ld-linux.so.2 (0x00ed6000)
> >
> > Although the fuse_dfs_wrapper.sh script has specified the LD_LIBRARY_PATH
> > there, I had to do an explicit export of the variable so that
> libhdfs.so.0
> > and libjvm.so files above had links to the correct object files.
> >
> > Could you please tell what could the problem be now?
> >
> > Thanks,
> >
> > Aastha.
> >
> > On 18 March 2011 03:08, Brian Bockelman <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Aastha,
> >>
> >> Try using "ldd" against the fuse_dfs executable, and see where you are
> >> pulling libhdfs.so from.  It may be it is linking from the "wrong one".
> >>
> >> Brian
> >>
> >> On Mar 17, 2011, at 3:24 PM, Aastha Mehta wrote:
> >>
> >>> Hello,
> >>>
> >>> I am working on a project involving hdfs and fuse-dfs API on top of it.
> I
> >>> wanted to trace through the functions called from libhdfs API by
> fuse-dfs
> >>> functions. I added print statements inside the hdfs.c file in
> appropriate
> >>> places to see how the functions progress. I execute ant
> >> compile-c++-libhdfs
> >>> -Dlibhdfs=1 and then ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
> >>> -Djava5.home=/usr/lib/jvm/java-1.5.0-sun. However, when I use fuse-dfs
> I
> >>> cannot see any of the print statements executed from libhdfs/hdfs.c.
> >>>
> >>> I am using hadoop-0.20.2 version and the libhdfs is present in
> >>> hadoop-0.20.2/src/c++/libhdfs. Could someone tell me if this libhdfs is
> >> the
> >>> one compiled and used or if there will be some other libhdfs that is
> >>> accessed. If this is the one, then why are the changes made in its
> files
> >>> reflected on running the code?
> >>>
> >>> Thanks,
> >>> Aastha.
> >>>
> >>> --
> >>> Aastha Mehta
Aastha Mehta
Intern, NetApp, Bangalore
4th year undergraduate, BITS Pilani
E-mail: [EMAIL PROTECTED]