Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> Fwd: problems with fuse-dfs


Copy link to this message
-
Fwd: problems with fuse-dfs
Sorry, resending on hdfs-dev; apparently I'm not on -user.

Begin forwarded message:

> From: Brian Bockelman <[EMAIL PROTECTED]>
> Date: March 1, 2011 6:24:28 PM CST
> Cc: hdfs-user <[EMAIL PROTECTED]>
> Subject: Re: problems with fuse-dfs
>
>
> Side note: Do not cross post to multiple lists.  It annoys folks.
>
> On Mar 1, 2011, at 11:50 AM, Aastha Mehta wrote:
>
>> Hello,
>>
>> I am facing problems in running fuse-dfs over hdfs. I came across this
>> thread while searching for my problem:
>> http://www.mail-archive.com/[EMAIL PROTECTED]/msg00341.html
>> OR
>> http://search-hadoop.com/m/T1Bjv17q0eF1&subj=Re+Fuse+DFS
>>
>> and it exactly mentions some of the symptoms I am looking at.
>>
>> To quote Eli,
>> "fuse_impls_getattr.c connects via hdfsConnectAsUser so you should see a log
>> (unless its returning from a case that doesn't print an error). Next
>> step is to determine that you're actually reaching the code you modified by
>> adding a syslog to the top of the function (need to make sure you're
>> actually loading the libhdfs you've built vs an older one or another one
>> installed on your system), and then determine which error case in that
>> function you're seeing. It's strange that -d would cause that path to
>> change."
>>
>> I get the error from fuse_impls_getattr.c that could not connect to my host
>> and port. I checked the syslogs and only this error is printed there. No
>> error is printed from hdfsConnectAsUser(). I even wrote down syslog
>> functions in hdfsConnectAsUser, but it is not printed. I used the following
>> commands to compile libhdfs and then fuse-dfs from $HADOOP_HOME:
>>
>> $ export
>> LD_LIBRARY_PATH=$HADOOP_HOME/build/libhdfs:/usr/lib/jvm/java-1.5.0-sun:
>> /usr/lib/jvm/java-1.5.0-sun-1.5.0.19/include:/usr/lib/jvm/java-6-sun-1.6.0.22/jre/lib/i386/client
>> $ ant compile-c++-libhdfs -Dlibhdfs=1
>> $ ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
>> -Djava5.home=/usr/lib/jvm/java-1.5.0-sun-1.5.0.19
>> $ mkdir /media/myDrive/newhdfs
>> $ cd $HADOOP_HOME/build/contrib/fuse-dfs/
>> $ sudo sh fuse_dfs_wrapper.sh dfs://aastha-desktop:9000
>> /media/myDrive/newhdfs
>>
>> And when i do:
>> $ sudo cat /var/log/syslog
>>
>> at the end, I get
>> aastha-desktop fuse_dfs: ERROR: could not connect to aastha-desktop:9000,
>> fuse_impls_getattr.c:37
>>
>> EIO value is 5, which I understand implies an Input/output error.
>>
>> When i try to ls /media/myDrive/newhdfs, i get the input/output error.
>> Also if i try to unmount newhdfs, it says that it is in use and cannot be
>> unmounted.
>>
>> I even tried -d option, but that just runs into an infinite loop of
>> connection retries and shows some socketconnection exception.
>>
>> I have been working with fuse-dfs and hdfs for more than a month now. Till
>> two days back I could atleast create fuse-dfs instances and mount them using
>> the wrapper script. I could also create files inside it, modify them, etc.
>> and view the changes being updated in the nodes of the hadoop cluster. Since
>> yesterday, I am trying to create a LUN image inside fuse-dfs (using dd
>> if=/dev/zero of=... command), and check whether it is created properly on
>> the cluster or not. Yesterday, a normal file system was working and when I
>> tried to create a LUN, my syslog showed that an error was coming from
>> fuse_impls_open.c. But now a normal file system instance is also not getting
>> created and mounted properly.
>>
>> Following are my questions based on this observation:
>> 1. Please tell me, why is the fuse_impls_getattr.c error is coming now and
>> how can it be resolved.
>
> Did you try adding "-d" to the options to run in debug mode?
>
> What is the resulting options for "fuse_dfs" if you do "ps faux"?
>
>> 2. Does ant compile-c++-libhdfs compile libhdfs? Will the changes that I
>> make in source files in libhdfs be updated by the command? Since I can't see
>> any syslog statements getting printed from hdfsConnectAsUser(), I am not
>> sure if libhdfs has been recompiled. Also I cannot see any object files
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB