Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Need you to check if Flume source/sink configuration is ok?


Copy link to this message
-
RE: Need you to check if Flume source/sink configuration is ok?
As advised, I tried to configure the same node to send data to the active collector. I did the following changes via Flume Master Web console:

[cid:[EMAIL PROTECTED]2FE0]
When I press "Submit Query", and I check the Flume logs ->

okeeper/zookeeper.jar:/usr/lib/flume/build/classes
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/flume/lib::/usr/lib/hadoop/lib/native/Linux-amd64-64
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-131.17.1.el6.x86_64
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=flume
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/var/run/flume
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/usr/lib/flume
2012-12-17 01:59:28,278 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
2012-12-17 01:59:28,315 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181
2012-12-17 01:59:28,333 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
2012-12-17 01:59:28,351 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13b82b043240027, negotiated timeout = 40000
2012-12-17 01:59:30,673 INFO org.terracotta.modules.ehcache.store.ClusteredStore: Cache [RTLogCache] using concurrency: 256
2012-12-17 01:59:30,896 INFO net.sf.ehcache.pool.sizeof.JvmInformation: Detected JVM data model settings of: 64-Bit HotSpot JVM with Compressed OOPs
2012-12-17 01:59:31,238 INFO net.sf.ehcache.pool.sizeof.AgentLoader: Extracted agent jar to temporary file /tmp/ehcache-sizeof-agent5050905318542369702.jar
2012-12-17 01:59:31,238 INFO net.sf.ehcache.pool.sizeof.AgentLoader: Trying to load agent @ /tmp/ehcache-sizeof-agent5050905318542369702.jar
2012-12-17 01:59:31,244 INFO net.sf.ehcache.pool.impl.DefaultSizeOfEngine: using Agent sizeof engine
2012-12-17 01:59:31,273 INFO net.sf.ehcache.pool.impl.DefaultSizeOfEngine: using Agent sizeof engine
2012-12-17 02:26:47,938 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 1 heartbeats
2012-12-17 02:26:52,939 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 2 heartbeats
2012-12-17 02:26:57,941 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 3 heartbeats
2012-12-17 02:27:02,943 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 4 heartbeats
2012-12-17 02:27:07,942 ERROR com.cloudera.flume.agent.LogicalNode: Forcing driver to exit uncleanly
2012-12-17 02:27:07,943 ERROR com.cloudera.flume.core.connector.DirectDriver: Closing down due to exception during append calls
2012-12-17 02:27:07,943 INFO com.cloudera.flume.agent.LogicalNode: Node config successfully set to FlumeConfigData: {srcVer:'Mon Dec 17 02:26:37 EST 2012' snkVer:'Mon Dec 17 02:26:37 EST 2012'  ts='Mon Dec 17 02:26:37 EST 2012' flowId:'default-flow' source:'tail( "/var/log/flume/test1.log", true )' sink:'{ value( "LogType", "Test1Log" ) => agentBESink( "ip-10-40-222-77.ec2.internal", 35853 ) }' }
2012-12-17 02:27:07,944 INFO com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode ec2-75-101-165-36.compute-1.amazonaws.com-19 exited with error: Waiting for queue element was interrupted! null
java.io.IOException: Waiting for queue element was interrupted! null
        at com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:222)
        at com.cloudera.flume.collector.CollectorSource.next(CollectorSource.java:72)
        at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:108)
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2038)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
        at com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:209)
        ... 2 more
2012-12-17 02:27:07,944 INFO com.cloudera.flume.collector.CollectorSource: closed
2012-12-17 02:27:07,946 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Closed server on port 35853...
2012-12-17 02:27:07,948 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Queue still has 0 elements ...
2012-12-17 02:27:07,949 ERROR com.cloudera.flume.core.connector.DirectDriver: Exiting driver logicalNode ec2-75-101-165-36.compute-1.amazonaws.com-19 in error state CollectorSource | EHCacheSink because Waiting for queue element was interrupted! null
2012-12-17 02:27:07,952 INFO com.cloudera.flume.handlers.debug.InsistentOpenDecorator: open attempt 0 failed, backoff (1000ms): Failed to open thrift event sink to ip-10-40-222-77.ec2.internal:35853 : java.net.ConnectException: Connection refused
2012-12-17 02:27:08,953 INFO com.cloudera.flume.handlers.debug.InsistentOpenDecorator: open attempt 1 failed, backoff (2000ms): Failed to open thrift event sink to ip-10-40-222-77.ec2.internal:35853 : java.net.ConnectException: Connection refused
2012-12-17 02:27:10,953 INFO com.cloudera.flume.handlers.debug.I
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB