Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Flume Source and Sink in different hosts


+
Kumar, Suresh 2012-10-04, 21:53
+
Hari Shreedharan 2012-10-04, 22:02
+
Kumar, Suresh 2012-10-04, 22:19
+
Hari Shreedharan 2012-10-04, 22:40
+
Kumar, Suresh 2012-10-04, 22:46
Copy link to this message
-
Re: Flume Source and Sink in different hosts
It depends on what kind of guarantees you need. If you need to make sure your events are persisted even during process/system failures, you should use the File Channel, else you can use Memory Channel (performance of Memory Channel is obviously better).  

Thanks,
Hari

--  
Hari Shreedharan
On Thursday, October 4, 2012 at 3:46 PM, Kumar, Suresh wrote:

>  
> Hari, I just noticed some entries in HBase, so this configuration does work.
> I will retry with the changes you recommended. Do you think I should be using
> some other channel type instead of memory?
>  
> Thanks,
> Suresh
> From: Hari Shreedharan [mailto:[EMAIL PROTECTED]]  
> Sent: Thursday, October 04, 2012 3:40 PM
> To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> Subject: Re: Flume Source and Sink in different hosts
>  
> Looks like your agent was set up properly. Can you increase the heap and try again? You can do this by setting -Xmx in the flume-env.sh (http://flume-env.sh) file. Try setting it to 1G or higher, since you are using memory channel. Also I assume the file you are tailing is getting written to? I strongly suggest using the AsyncHBaseSink.  
>  
>  
>  
>  
>  
> Thanks,
>  
> Hari
>  
>  
>  
> --  
>  
> Hari Shreedharan
>  
>  
>  
>  
> On Thursday, October 4, 2012 at 3:19 PM, Kumar, Suresh wrote:
> >  
> > Yes, my HBase has the table and column family, if I run the /etc/passwd test using flume-ng client, the table
> >  
> >  
> > gets populated.
> >  
> >  
> >  
> >  
> >  
> > Here is the log from the source agent, there is nothing much in the sink except for which seem to benign.
> >  
> >  
> >  
> > Thanks,
> >  
> >  
> > Suresh
> >  
> >  
> >  
> >  
> >  
> > 2012-10-04 14:59:05,622 (lifecycleSupervisor-1-0-SendThread(localhost:2181)) [DEBUG - org.apache.zookeeper.client.ZooKeeperSaslClient.clientTunneledAuthenticationInProgress(ZooKeeperSaslClient.java:515)] Could not retrieve login configuration: java.lang.SecurityException: Unable to locate a login configuration
> >  
> >  
> > 2012-10-04 14:59:08,414 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)] Checking file:conf/flume.conf (file:///\\conf\flume.conf) for changes
> >  
> >  
> >  
> >  
> >  
> > source agent log:
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> >  
> > $ bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent3
> >  
> >  
> >  
> >  
> >  
> > + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/opt/flume/conf:/opt/flume/lib/*' -Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf -n agent3
> >  
> >  
> > 2012-10-04 15:09:30,778 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
> >  
> >  
> > 2012-10-04 15:09:30,791 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent3
> >  
> >  
> > 2012-10-04 15:09:30,799 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:203)] Node manager starting
> >  
> >  
> > 2012-10-04 15:09:30,801 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
> >  
> >  
> > 2012-10-04 15:09:30,810 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 10
> >  
> >  
> > 2012-10-04 15:09:30,813 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:207)] Node manager started
> >  
> >  
> > 2012-10-04 15:09:30,819 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)] Configuration provider started
+
Kumar, Suresh 2012-10-05, 18:07
+
Kumar, Suresh 2012-10-05, 18:27
+
Hari Shreedharan 2012-10-05, 18:40
+
Kumar, Suresh 2012-10-05, 21:55
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB