Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: How to support the (HDFS) FileSystem API of various Hadoop Distributions?


Copy link to this message
-
Re: How to support the (HDFS) FileSystem API of various Hadoop Distributions?
Hi,

I think the simpler way is to distribute multiple pre-built packages,
targeting each version/distribution, instead of trying to detect which
one to load?

On Sat, Sep 7, 2013 at 4:14 PM, Christian Schneider
<[EMAIL PROTECTED]> wrote:
> Hi Harsh,
> Thanks for the suggestion. And yes, the .isRoot() lead to this question :).
> Also I wasn't sure about the ideal way of packaging this utility.
>
> My "dream" is to have 1 binary that is able to deal with different versions
> (IPC 3, 5, 7, ...).
> The users will download the binary package and it is compatible with a wide
> range of versions.
>
> From your suggestions I tried to run 0.20.2 against CDH4.3, - but it is not
> compatible with IPC V7.
>
> So I think I need to do the same then HBase Compat does. "Somehow" the tool
> needs to check
> the version of the cluster and then load the correct implementation for
> that.
>
> But how to check the IPC version?
>
> Best Regards,
> Christian.
>
>
> P.S.: Thanks, that motivates me to continue :)
>
>
> 2013/9/6 Harsh J <[EMAIL PROTECTED]>
>>
>> Oh and btw, nice utility! :)
>>
>> On Fri, Sep 6, 2013 at 7:50 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>> > Hello,
>> >
>> > There are a few additions to the FileSystem that may bite you across
>> > versions, but if you pick an old stable version such as Apache Hadoop
>> > 0.20.2, and stick to only its offered APIs, it would work better
>> > across different version dependencies as we try to maintain FileSystem
>> > as a stable interface as much as we can (there was also more recent
>> > work to ensure the stabilization). I looked over your current code
>> > state and it seemed to have pretty stable calls that I think have
>> > existed across several versions and exists today, but I did notice you
>> > had to remove an isRoot as part of a previous commit, which may have
>> > lead to this question?
>> >
>> > If that doesn't work for you, you can also switch out to using
>> > sub-modules carrying code specific to a build version type (such as
>> > what HBase does at https://github.com/apache/hbase/tree/trunk/ (see
>> > the hbase-hadoop-compat directories)).
>> >
>> > On Fri, Sep 6, 2013 at 2:59 AM, Christian Schneider
>> > <[EMAIL PROTECTED]> wrote:
>> >> Hi,
>> >> I start to write a small ncdu clone to browse HDFS on the CLI
>> >> (http://nchadoop.org/). Currently i'm testing it against CDH4, - but I
>> >> like
>> >> to make it available for a wider group of users (Hortonworks, ..).
>> >>
>> >> Is it enough to pick different vanilla Versions (for IPC 5, 7)?
>> >>
>> >> Best Regards,
>> >> Christian.
>> >>
>> >
>> >
>> >
>> > --
>> > Harsh J
>>
>>
>>
>> --
>> Harsh J
>
>

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB