Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Flume Source and Sink in different hosts


Copy link to this message
-
Re: Flume Source and Sink in different hosts
Hari Shreedharan 2012-10-04, 23:25
It depends on what kind of guarantees you need. If you need to make sure your events are persisted even during process/system failures, you should use the File Channel, else you can use Memory Channel (performance of Memory Channel is obviously better).  

Thanks,
Hari

--  
Hari Shreedharan
On Thursday, October 4, 2012 at 3:46 PM, Kumar, Suresh wrote:

>  
> Hari, I just noticed some entries in HBase, so this configuration does work.
> I will retry with the changes you recommended. Do you think I should be using
> some other channel type instead of memory?
>  
> Thanks,
> Suresh
> From: Hari Shreedharan [mailto:[EMAIL PROTECTED]]  
> Sent: Thursday, October 04, 2012 3:40 PM
> To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> Subject: Re: Flume Source and Sink in different hosts
>  
> Looks like your agent was set up properly. Can you increase the heap and try again? You can do this by setting -Xmx in the flume-env.sh (http://flume-env.sh) file. Try setting it to 1G or higher, since you are using memory channel. Also I assume the file you are tailing is getting written to? I strongly suggest using the AsyncHBaseSink.  
>  
>  
>  
>  
>  
> Thanks,
>  
> Hari
>  
>  
>  
> --  
>  
> Hari Shreedharan
>  
>  
>  
>  
> On Thursday, October 4, 2012 at 3:19 PM, Kumar, Suresh wrote:
> >  
> > Yes, my HBase has the table and column family, if I run the /etc/passwd test using flume-ng client, the table
> >  
> >  
> > gets populated.
> >  
> >  
> >  
> >  
> >  
> > Here is the log from the source agent, there is nothing much in the sink except for which seem to benign.
> >  
> >  
> >  
> > Thanks,
> >  
> >  
> > Suresh
> >  
> >  
> >  
> >  
> >  
> > 2012-10-04 14:59:05,622 (lifecycleSupervisor-1-0-SendThread(localhost:2181)) [DEBUG - org.apache.zookeeper.client.ZooKeeperSaslClient.clientTunneledAuthenticationInProgress(ZooKeeperSaslClient.java:515)] Could not retrieve login configuration: java.lang.SecurityException: Unable to locate a login configuration
> >  
> >  
> > 2012-10-04 14:59:08,414 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)] Checking file:conf/flume.conf (file:///\\conf\flume.conf) for changes
> >  
> >  
> >  
> >  
> >  
> > source agent log:
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > $ bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent3
> >  
> >  
> >  
> >  
> >  
> > + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/opt/flume/conf:/opt/flume/lib/*' -Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf -n agent3
> >  
> >  
> > 2012-10-04 15:09:30,778 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
> >  
> >  
> > 2012-10-04 15:09:30,791 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent3
> >  
> >  
> > 2012-10-04 15:09:30,799 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:203)] Node manager starting
> >  
> >  
> > 2012-10-04 15:09:30,801 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
> >  
> >  
> > 2012-10-04 15:09:30,810 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 10
> >  
> >  
> > 2012-10-04 15:09:30,813 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:207)] Node manager started
> >  
> >  
> > 2012-10-04 15:09:30,819 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)] Configuration provider started