Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Flume Source and Sink in different hosts


+
Kumar, Suresh 2012-10-04, 21:53
+
Hari Shreedharan 2012-10-04, 22:02
+
Kumar, Suresh 2012-10-04, 22:19
Copy link to this message
-
Re: Flume Source and Sink in different hosts
Hari Shreedharan 2012-10-04, 22:40
Looks like your agent was set up properly. Can you increase the heap and try again? You can do this by setting -Xmx in the flume-env.sh file. Try setting it to 1G or higher, since you are using memory channel. Also I assume the file you are tailing is getting written to? I strongly suggest using the AsyncHBaseSink.  
Thanks,
Hari

--  
Hari Shreedharan
On Thursday, October 4, 2012 at 3:19 PM, Kumar, Suresh wrote:

> Yes, my HBase has the table and column family, if I run the /etc/passwd test using flume-ng client, the table
> gets populated.
>  
> Here is the log from the source agent, there is nothing much in the sink except for which seem to benign.
>  
> Thanks,
> Suresh
>  
> 2012-10-04 14:59:05,622 (lifecycleSupervisor-1-0-SendThread(localhost:2181)) [DEBUG - org.apache.zookeeper.client.ZooKeeperSaslClient.clientTunneledAuthenticationInProgress(ZooKeeperSaslClient.java:515)] Could not retrieve login configuration: java.lang.SecurityException: Unable to locate a login configuration
> 2012-10-04 14:59:08,414 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)] Checking file:conf/flume.conf for changes
>  
> source agent log:
>  
>  
> $ bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent3
>  
> + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/opt/flume/conf:/opt/flume/lib/*' -Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf -n agent3
> 2012-10-04 15:09:30,778 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
> 2012-10-04 15:09:30,791 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent3
> 2012-10-04 15:09:30,799 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:203)] Node manager starting
> 2012-10-04 15:09:30,801 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
> 2012-10-04 15:09:30,810 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 10
> 2012-10-04 15:09:30,813 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:207)] Node manager started
> 2012-10-04 15:09:30,819 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)] Configuration provider started
> 2012-10-04 15:09:30,819 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)] Checking file:conf/flume.conf for changes
> 2012-10-04 15:09:30,821 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:conf/flume.conf
> 2012-10-04 15:09:30,839 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:912)] Added sinks: avro-sink Agent: agent3
> 2012-10-04 15:09:30,840 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)] Processing:avro-sink
> 2012-10-04 15:09:30,840 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1002)] Created context for avro-sink: hostname
> 2012-10-04 15:09:30,841 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:998)] Processing:avro-sink
+
Kumar, Suresh 2012-10-04, 22:46
+
Hari Shreedharan 2012-10-04, 23:25
+
Kumar, Suresh 2012-10-05, 18:07
+
Kumar, Suresh 2012-10-05, 18:27
+
Hari Shreedharan 2012-10-05, 18:40
+
Kumar, Suresh 2012-10-05, 21:55