Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> RDMA (InfiniBand and RoCE) Support for Hadoop 1.2.1


Copy link to this message
-
Re: RDMA (InfiniBand and RoCE) Support for Hadoop 1.2.1
Hey, Milind,
Thanks for your interests.

- We are working on moving the design to Hadoop 2.x. It will be
available in the near future.

- This is an independent project.

Xiaoyi

On Mon, Nov 11, 2013 at 2:43 PM, Milind Bhandarkar
<[EMAIL PROTECTED]> wrote:
> Hi Xiaoyi,
>
> This is great !
>
> For clusters that have moved to Hadoop 2.x versions, do you have any work
> in progress to rebase Hadoop-RDMA to 2.x ?
>
> Also, I am wondering if the development that you are doing will be
> available in Mellanox UDA ? Or is it an independent project ?
>
> - milind
>
>
> ---
> Milind Bhandarkar
> Chief Scientist
> Pivotal
> +1-650-523-3858 (W)
> +1-408-666-8483 (C)
>
>
> On Sat, Nov 9, 2013 at 5:44 AM, Xiaoyi [EMAIL PROTECTED] <
> [EMAIL PROTECTED]> wrote:
>
>> The Hadoop-RDMA team is pleased to announce the release of Hadoop-RDMA
>> 0.9.8
>> (technology preview) package with the following features and bug-fixes:
>>
>> * Hadoop-RDMA 0.9.8 Features
>>
>>    - Based on Apache Hadoop 1.2.1
>>    - High performance design with native InfiniBand and RoCE support
>>      at the verbs-level for HDFS, MapReduce, and RPC components
>>    - Compliant with Apache Hadoop 1.2.1 APIs and applications
>>    - Easily configurable for native InfiniBand, RoCE and the traditional
>>      sockets-based support (Ethernet and InfiniBand with IPoIB)
>>    - HDFS over native InfiniBand and RoCE
>>        - RDMA-based write
>>        - RDMA-based replication
>>        - Tuned packet size
>>    - MapReduce over native InfiniBand and RoCE
>>        - RDMA-based shuffle
>>        - Pre-fetching and caching of map outputs
>>        - In-memory merge
>>        - Overlapping of merge and reduce
>>    - RPC over native InfiniBand and RoCE
>>        - JVM-bypassed buffer management
>>        - On-demand InfiniBand connection setup
>>        - RDMA or send/recv based adaptive communication
>>    - Tested with
>>        - Mellanox InfiniBand adapters (DDR, QDR, and FDR)
>>        - RoCE
>>        - Various multi-core platforms
>>        - Different file systems with disks and SSDs
>>
>> * Bug Fixes (since 0.9.1)
>>
>>    - MapReduce over InfiniBand
>>      - Reducing the waiting time for the pre-fetch cache read
>>      - Enabling Reduce tasks to randomize the shuffle pattern
>>    - HDFS over InfiniBand
>>      - Fix an issue in processing multiple blocks concurrently in a
>>        single task
>>
>> For downloading Hadoop-RDMA 0.9.8 package and the associated user guide,
>> please visit the following URL:
>>
>> http://hadoop-rdma.cse.ohio-state.edu
>>
>> Sample performance numbers for benchmarks using Hadoop-RDMA 0.9.8
>> version can be viewed by visiting the `Performance' tab of the above
>> website.
>>
>> All questions, feedbacks and bug reports are welcome. Please post it
>> to the hadoop-rdma-discuss mailing list (hadoop-rdma-discuss at
>> cse.ohio-state.edu).
>>
>> Thanks,
>>
>> The Hadoop-RDMA Team
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB