Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: How to support the (HDFS) FileSystem API of various Hadoop Distributions?


Copy link to this message
-
Re: How to support the (HDFS) FileSystem API of various Hadoop Distributions?
Hi,

I think the simpler way is to distribute multiple pre-built packages,
targeting each version/distribution, instead of trying to detect which
one to load?

On Sat, Sep 7, 2013 at 4:14 PM, Christian Schneider
<[EMAIL PROTECTED]> wrote:
> Hi Harsh,
> Thanks for the suggestion. And yes, the .isRoot() lead to this question :).
> Also I wasn't sure about the ideal way of packaging this utility.
>
> My "dream" is to have 1 binary that is able to deal with different versions
> (IPC 3, 5, 7, ...).
> The users will download the binary package and it is compatible with a wide
> range of versions.
>
> From your suggestions I tried to run 0.20.2 against CDH4.3, - but it is not
> compatible with IPC V7.
>
> So I think I need to do the same then HBase Compat does. "Somehow" the tool
> needs to check
> the version of the cluster and then load the correct implementation for
> that.
>
> But how to check the IPC version?
>
> Best Regards,
> Christian.
>
>
> P.S.: Thanks, that motivates me to continue :)
>
>
> 2013/9/6 Harsh J <[EMAIL PROTECTED]>
>>
>> Oh and btw, nice utility! :)
>>
>> On Fri, Sep 6, 2013 at 7:50 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>> > Hello,
>> >
>> > There are a few additions to the FileSystem that may bite you across
>> > versions, but if you pick an old stable version such as Apache Hadoop
>> > 0.20.2, and stick to only its offered APIs, it would work better
>> > across different version dependencies as we try to maintain FileSystem
>> > as a stable interface as much as we can (there was also more recent
>> > work to ensure the stabilization). I looked over your current code
>> > state and it seemed to have pretty stable calls that I think have
>> > existed across several versions and exists today, but I did notice you
>> > had to remove an isRoot as part of a previous commit, which may have
>> > lead to this question?
>> >
>> > If that doesn't work for you, you can also switch out to using
>> > sub-modules carrying code specific to a build version type (such as
>> > what HBase does at https://github.com/apache/hbase/tree/trunk/ (see
>> > the hbase-hadoop-compat directories)).
>> >
>> > On Fri, Sep 6, 2013 at 2:59 AM, Christian Schneider
>> > <[EMAIL PROTECTED]> wrote:
>> >> Hi,
>> >> I start to write a small ncdu clone to browse HDFS on the CLI
>> >> (http://nchadoop.org/). Currently i'm testing it against CDH4, - but I
>> >> like
>> >> to make it available for a wider group of users (Hortonworks, ..).
>> >>
>> >> Is it enough to pick different vanilla Versions (for IPC 5, 7)?
>> >>
>> >> Best Regards,
>> >> Christian.
>> >>
>> >
>> >
>> >
>> > --
>> > Harsh J
>>
>>
>>
>> --
>> Harsh J
>
>

--
Harsh J