Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: Ability to specify the capacity of MemoryChannel in bytes


+
Ted Malaska 2012-09-10, 13:59
+
Mike Percy 2012-09-14, 06:14
+
Ted Malaska 2012-09-14, 14:00
+
Mike Percy 2012-09-14, 20:44
+
Ted Malaska 2012-09-14, 20:45
+
Ted Malaska 2012-09-14, 20:48
+
Ted Malaska 2012-09-14, 21:14
+
Hari Shreedharan 2012-09-15, 21:21
+
Ted Malaska 2012-09-15, 22:20
+
Hari Shreedharan 2012-09-15, 22:31
+
Ted Malaska 2012-09-16, 03:47
+
Ted Malaska 2012-09-19, 15:28
+
Ted Malaska 2012-09-16, 03:54
+
Hari Shreedharan 2012-10-01, 20:30
+
Hari Shreedharan 2012-10-01, 20:32
+
Hari Shreedharan 2012-10-01, 20:51
Copy link to this message
-
Re: Review Request: Ability to specify the capacity of MemoryChannel in bytes
Ted Malaska 2012-10-02, 00:43


> On Oct. 1, 2012, 8:30 p.m., Hari Shreedharan wrote:
> > flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java, lines 60-64
> > <https://reviews.apache.org/r/6982/diff/3/?file=155699#file155699line60>
> >
> >     This is still not thread safe. The transaction objects are thread local. SO if you have bufferedByteCapacity = 10 and 2 txns doing puts, one where the size = 3 and the other where size = 8. This series of events happens:
> >    
> >     txn 1 increments currentByteUsage atomically setting it to 3.
> >     Before txn1 finishes the comparison txn 2 adds 8 to currentByteUsage and does the comparison, and gets true, enters the if block.
> >     Now txn1 does the compare, and gets true - and enters the if block.
> >    
> >     Both txn1 and txn2 fail even though one of the two could easily have gone in.
> >    
> >     You need to make sure the comparison is atomic.
>
> Hari Shreedharan wrote:
>     This sort of threading issues can be easily fixed by using semaphores.

I'm new to semaphores.  If you set the number of permits to 1 how is it different from synchronized?
> On Oct. 1, 2012, 8:30 p.m., Hari Shreedharan wrote:
> > flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java, lines 122-126
> > <https://reviews.apache.org/r/6982/diff/3/?file=155699#file155699line122>
> >
> >     These have threading and efficiency implications.
> >    
> >     Since these are inside transactions, wouldn't this be much more efficient and thread-safe if you just keep track of the size internally per transaction, rather than sitting in a loop and subtracting ?

agreed
> On Oct. 1, 2012, 8:30 p.m., Hari Shreedharan wrote:
> > flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java, lines 153-156
> > <https://reviews.apache.org/r/6982/diff/3/?file=155699#file155699line153>
> >
> >     same as above.

agreed
- Ted
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6982/#review12075
-----------------------------------------------------------
On Sept. 16, 2012, 3:54 a.m., Ted Malaska wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6982/
> -----------------------------------------------------------
>
> (Updated Sept. 16, 2012, 3:54 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> 1. The user will be able to define a byteCapacity and a byteCapacityBufferPercentage.
> 2. Events byte size will be estimated from there body contents
> 3. On put bytes are added to current total
> 4. On commit any uncommitted takes are removed from the current byte total
> 5. On rollover any uncommitted puts are removed from the current byte total
>
>
> This addresses bug FLUME-1535.
>     https://issues.apache.org/jira/browse/FLUME-1535
>
>
> Diffs
> -----
>
>   flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java c72e97c
>   flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
>
> Diff: https://reviews.apache.org/r/6982/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ted Malaska
>
>

+
Hari Shreedharan 2012-10-02, 00:48
+
Hari Shreedharan 2012-10-02, 00:50
+
Ted Malaska 2012-10-02, 11:31
+
Hari Shreedharan 2012-10-02, 17:36
+
Hari Shreedharan 2012-10-02, 21:07
+
Ted Malaska 2012-10-03, 00:39
+
Ted Malaska 2012-10-03, 10:37
+
Ted Malaska 2012-10-04, 03:33
+
Ted Malaska 2012-10-04, 11:44
+
Ted Malaska 2012-10-04, 11:45
+
Ted Malaska 2012-10-04, 11:52
+
Hari Shreedharan 2012-10-04, 19:16
+
Ted Malaska 2012-10-04, 23:47
+
Ted Malaska 2012-10-04, 23:47
+
Ted Malaska 2012-10-05, 11:55
+
Hari Shreedharan 2012-10-05, 22:23