Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> File Channel Capacity issue


+
Camp, Roy 2012-11-23, 19:58
Copy link to this message
-
Re: File Channel Capacity issue
Roy,  

The FileChannel actually uses a fixed size checkpoint file -- so it is not possible to set it to unlimited size (the checkpoint file is mmap-ed to a fixed size buffer). To change the capacity of the channel, the easiest way off the top of my head is:

* Shutdown the agent.
* Delete all files in the file channel's checkpoint directory. (not the data directories. Also you might want to move them out, rather than delete to be safe)
* Change your configuration to increase the capacity of the channel.
* Restart the agent - this will cause full replay, so the agent might take sometime to start up if there are a lot of events in the channel (to avoid this - shutdown the source before shutting the agent down - so the sink can drain out the channel completely, wait for about 1-2 mins after the channel is empty so that the data files get deleted (this happens only immediately after a checkpoint - you can verify this by making sure each data dir has only 2 files each), since all events have been sent out - so during restart the channel will be quite empty, with very little to replay).
Hope this helps.

Thanks,
Hari

--  
Hari Shreedharan
On Friday, November 23, 2012 at 11:58 AM, Camp, Roy wrote:

> I am having issue with a slow sink (not flume related) but it is causing my file channel to overflow.  When trying to increase it I get the following error but can’t seem to find the setting it refers to in any of the documentation.  Additionally, is there a way to set the file channel to unlimited?  It doesn’t seem to like 0.  
>  
> java.lang.IllegalStateException: Channel closed [channel=collectorfile]. Due to java.lang.IllegalStateException: Configured capacity is 100000000 but the  checkpoint file capacity is 1000000. See FileChannel documentation on how to change a channels capacity.
>  
>  
> Thanks,
>  
> Roy
>  
>  
>  
>  
>  
+
Camp, Roy 2012-11-25, 23:08
+
Brock Noland 2012-11-25, 23:13
+
Camp, Roy 2012-11-26, 23:29
+
Brock Noland 2012-11-26, 23:57