-Fwd: problems with fuse-dfs
Brian Bockelman 2011-03-02, 00:26
Sorry, resending on hdfs-dev; apparently I'm not on -user.
Begin forwarded message:
> From: Brian Bockelman <[EMAIL PROTECTED]>
> Date: March 1, 2011 6:24:28 PM CST
> Cc: hdfs-user <[EMAIL PROTECTED]>
> Subject: Re: problems with fuse-dfs
> Side note: Do not cross post to multiple lists. It annoys folks.
> On Mar 1, 2011, at 11:50 AM, Aastha Mehta wrote:
>> I am facing problems in running fuse-dfs over hdfs. I came across this
>> thread while searching for my problem:
>> http://www.mail-archive.com/[EMAIL PROTECTED]/msg00341.html
>> and it exactly mentions some of the symptoms I am looking at.
>> To quote Eli,
>> "fuse_impls_getattr.c connects via hdfsConnectAsUser so you should see a log
>> (unless its returning from a case that doesn't print an error). Next
>> step is to determine that you're actually reaching the code you modified by
>> adding a syslog to the top of the function (need to make sure you're
>> actually loading the libhdfs you've built vs an older one or another one
>> installed on your system), and then determine which error case in that
>> function you're seeing. It's strange that -d would cause that path to
>> I get the error from fuse_impls_getattr.c that could not connect to my host
>> and port. I checked the syslogs and only this error is printed there. No
>> error is printed from hdfsConnectAsUser(). I even wrote down syslog
>> functions in hdfsConnectAsUser, but it is not printed. I used the following
>> commands to compile libhdfs and then fuse-dfs from $HADOOP_HOME:
>> $ export
>> $ ant compile-c++-libhdfs -Dlibhdfs=1
>> $ ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
>> $ mkdir /media/myDrive/newhdfs
>> $ cd $HADOOP_HOME/build/contrib/fuse-dfs/
>> $ sudo sh fuse_dfs_wrapper.sh dfs://aastha-desktop:9000
>> And when i do:
>> $ sudo cat /var/log/syslog
>> at the end, I get
>> aastha-desktop fuse_dfs: ERROR: could not connect to aastha-desktop:9000,
>> EIO value is 5, which I understand implies an Input/output error.
>> When i try to ls /media/myDrive/newhdfs, i get the input/output error.
>> Also if i try to unmount newhdfs, it says that it is in use and cannot be
>> I even tried -d option, but that just runs into an infinite loop of
>> connection retries and shows some socketconnection exception.
>> I have been working with fuse-dfs and hdfs for more than a month now. Till
>> two days back I could atleast create fuse-dfs instances and mount them using
>> the wrapper script. I could also create files inside it, modify them, etc.
>> and view the changes being updated in the nodes of the hadoop cluster. Since
>> yesterday, I am trying to create a LUN image inside fuse-dfs (using dd
>> if=/dev/zero of=... command), and check whether it is created properly on
>> the cluster or not. Yesterday, a normal file system was working and when I
>> tried to create a LUN, my syslog showed that an error was coming from
>> fuse_impls_open.c. But now a normal file system instance is also not getting
>> created and mounted properly.
>> Following are my questions based on this observation:
>> 1. Please tell me, why is the fuse_impls_getattr.c error is coming now and
>> how can it be resolved.
> Did you try adding "-d" to the options to run in debug mode?
> What is the resulting options for "fuse_dfs" if you do "ps faux"?
>> 2. Does ant compile-c++-libhdfs compile libhdfs? Will the changes that I
>> make in source files in libhdfs be updated by the command? Since I can't see
>> any syslog statements getting printed from hdfsConnectAsUser(), I am not
>> sure if libhdfs has been recompiled. Also I cannot see any object files