Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> RDMA (InfiniBand and RoCE) Support for Hadoop 1.2.1


Copy link to this message
-
RDMA (InfiniBand and RoCE) Support for Hadoop 1.2.1
The Hadoop-RDMA team is pleased to announce the release of Hadoop-RDMA 0.9.8
(technology preview) package with the following features and bug-fixes:

* Hadoop-RDMA 0.9.8 Features

   - Based on Apache Hadoop 1.2.1
   - High performance design with native InfiniBand and RoCE support
     at the verbs-level for HDFS, MapReduce, and RPC components
   - Compliant with Apache Hadoop 1.2.1 APIs and applications
   - Easily configurable for native InfiniBand, RoCE and the traditional
     sockets-based support (Ethernet and InfiniBand with IPoIB)
   - HDFS over native InfiniBand and RoCE
       - RDMA-based write
       - RDMA-based replication
       - Tuned packet size
   - MapReduce over native InfiniBand and RoCE
       - RDMA-based shuffle
       - Pre-fetching and caching of map outputs
       - In-memory merge
       - Overlapping of merge and reduce
   - RPC over native InfiniBand and RoCE
       - JVM-bypassed buffer management
       - On-demand InfiniBand connection setup
       - RDMA or send/recv based adaptive communication
   - Tested with
       - Mellanox InfiniBand adapters (DDR, QDR, and FDR)
       - RoCE
       - Various multi-core platforms
       - Different file systems with disks and SSDs

* Bug Fixes (since 0.9.1)

   - MapReduce over InfiniBand
     - Reducing the waiting time for the pre-fetch cache read
     - Enabling Reduce tasks to randomize the shuffle pattern
   - HDFS over InfiniBand
     - Fix an issue in processing multiple blocks concurrently in a
       single task

For downloading Hadoop-RDMA 0.9.8 package and the associated user guide,
please visit the following URL:

http://hadoop-rdma.cse.ohio-state.edu

Sample performance numbers for benchmarks using Hadoop-RDMA 0.9.8
version can be viewed by visiting the `Performance' tab of the above website.

All questions, feedbacks and bug reports are welcome. Please post it
to the hadoop-rdma-discuss mailing list (hadoop-rdma-discuss at
cse.ohio-state.edu).

Thanks,

The Hadoop-RDMA Team
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB