Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Drill >> mail # dev >> Review Request 17165: Extend netty memory manager (jira DRILL-336)

John Morris 2014-01-22, 22:36
Jacques Nadeau 2014-01-23, 16:28
John Morris 2014-01-24, 23:16
Copy link to this message
Re: Review Request 17165: Extend netty memory manager (jira DRILL-336)

This is an automatically generated e-mail. To reply, visit:

(Updated Feb. 24, 2014, 11:46 p.m.)
Review request for drill.

Removed the "feature" of throwing an exception for large memory copy on resize. While we would like to avoid large copies in Drill, internal Netty routines were unwittingly raising the exception.
Repository: drill-git

Modified the netty direct memory manager to:
   1) Efficiently reduce ("trim") the size of a memory buffer, releasing the extra memory back to the memory manager.
   2) Opportunistically allocate larger buffers if it can be done efficiently.  
   3) Raise a warning exception if resizing a buffer copies more than 1 page of data.
   4) Added Javadocs and some in-code comments.
   5) Created a Junit test to verify basic functionality.

The "trim" and allocation changes are primarily in PoolChunkL,
which breaks a "chunk" into runs of pages using a buddy system.
Diffs (updated)

  exec/bufferl/src/main/java/io/netty/buffer/PoolArenaL.java cd9ec3f
  exec/bufferl/src/main/java/io/netty/buffer/PoolChunkL.java 91a13a9
  exec/bufferl/src/main/java/io/netty/buffer/PoolChunkListL.java cfa6fc6
  exec/bufferl/src/main/java/io/netty/buffer/PoolSubpageL.java 2956dd6
  exec/bufferl/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java f3b3afb
  exec/bufferl/src/main/java/io/netty/buffer/PooledByteBufL.java 595cbc9
  exec/bufferl/src/main/java/io/netty/buffer/PooledUnsafeDirectByteBufL.java efc13b5
  exec/bufferl/src/main/java/io/netty/buffer/package-info.java PRE-CREATION
  exec/bufferl/src/test/java/io/netty/buffer/TestPoolChunkTrim.java PRE-CREATION

Diff: https://reviews.apache.org/r/17165/diff/

Created a basic Junit test to verify the functionality as well as some of the boundary conditions.

John Morris