Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> RDMA (InfiniBand and RoCE) Support for Hadoop 1.2.1


Copy link to this message
-
Re: RDMA (InfiniBand and RoCE) Support for Hadoop 1.2.1
Hey, Milind,
Thanks for your interests.

- We are working on moving the design to Hadoop 2.x. It will be
available in the near future.

- This is an independent project.

Xiaoyi

On Mon, Nov 11, 2013 at 2:43 PM, Milind Bhandarkar
<[EMAIL PROTECTED]> wrote:
> Hi Xiaoyi,
>
> This is great !
>
> For clusters that have moved to Hadoop 2.x versions, do you have any work
> in progress to rebase Hadoop-RDMA to 2.x ?
>
> Also, I am wondering if the development that you are doing will be
> available in Mellanox UDA ? Or is it an independent project ?
>
> - milind
>
>
> ---
> Milind Bhandarkar
> Chief Scientist
> Pivotal
> +1-650-523-3858 (W)
> +1-408-666-8483 (C)
>
>
> On Sat, Nov 9, 2013 at 5:44 AM, Xiaoyi [EMAIL PROTECTED] <
> [EMAIL PROTECTED]> wrote:
>
>> The Hadoop-RDMA team is pleased to announce the release of Hadoop-RDMA
>> 0.9.8
>> (technology preview) package with the following features and bug-fixes:
>>
>> * Hadoop-RDMA 0.9.8 Features
>>
>>    - Based on Apache Hadoop 1.2.1
>>    - High performance design with native InfiniBand and RoCE support
>>      at the verbs-level for HDFS, MapReduce, and RPC components
>>    - Compliant with Apache Hadoop 1.2.1 APIs and applications
>>    - Easily configurable for native InfiniBand, RoCE and the traditional
>>      sockets-based support (Ethernet and InfiniBand with IPoIB)
>>    - HDFS over native InfiniBand and RoCE
>>        - RDMA-based write
>>        - RDMA-based replication
>>        - Tuned packet size
>>    - MapReduce over native InfiniBand and RoCE
>>        - RDMA-based shuffle
>>        - Pre-fetching and caching of map outputs
>>        - In-memory merge
>>        - Overlapping of merge and reduce
>>    - RPC over native InfiniBand and RoCE
>>        - JVM-bypassed buffer management
>>        - On-demand InfiniBand connection setup
>>        - RDMA or send/recv based adaptive communication
>>    - Tested with
>>        - Mellanox InfiniBand adapters (DDR, QDR, and FDR)
>>        - RoCE
>>        - Various multi-core platforms
>>        - Different file systems with disks and SSDs
>>
>> * Bug Fixes (since 0.9.1)
>>
>>    - MapReduce over InfiniBand
>>      - Reducing the waiting time for the pre-fetch cache read
>>      - Enabling Reduce tasks to randomize the shuffle pattern
>>    - HDFS over InfiniBand
>>      - Fix an issue in processing multiple blocks concurrently in a
>>        single task
>>
>> For downloading Hadoop-RDMA 0.9.8 package and the associated user guide,
>> please visit the following URL:
>>
>> http://hadoop-rdma.cse.ohio-state.edu
>>
>> Sample performance numbers for benchmarks using Hadoop-RDMA 0.9.8
>> version can be viewed by visiting the `Performance' tab of the above
>> website.
>>
>> All questions, feedbacks and bug reports are welcome. Please post it
>> to the hadoop-rdma-discuss mailing list (hadoop-rdma-discuss at
>> cse.ohio-state.edu).
>>
>> Thanks,
>>
>> The Hadoop-RDMA Team
>>