Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Need you to check if Flume source/sink configuration is ok?


Copy link to this message
-
RE: Need you to check if Flume source/sink configuration is ok?
As advised, I tried to configure the same node to send data to the active collector. I did the following changes via Flume Master Web console:

[cid:[EMAIL PROTECTED]2FE0]
When I press "Submit Query", and I check the Flume logs ->

okeeper/zookeeper.jar:/usr/lib/flume/build/classes
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/flume/lib::/usr/lib/hadoop/lib/native/Linux-amd64-64
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-131.17.1.el6.x86_64
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=flume
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/var/run/flume
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/usr/lib/flume
2012-12-17 01:59:28,278 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
2012-12-17 01:59:28,315 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181
2012-12-17 01:59:28,333 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
2012-12-17 01:59:28,351 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13b82b043240027, negotiated timeout = 40000
2012-12-17 01:59:30,673 INFO org.terracotta.modules.ehcache.store.ClusteredStore: Cache [RTLogCache] using concurrency: 256
2012-12-17 01:59:30,896 INFO net.sf.ehcache.pool.sizeof.JvmInformation: Detected JVM data model settings of: 64-Bit HotSpot JVM with Compressed OOPs
2012-12-17 01:59:31,238 INFO net.sf.ehcache.pool.sizeof.AgentLoader: Extracted agent jar to temporary file /tmp/ehcache-sizeof-agent5050905318542369702.jar
2012-12-17 01:59:31,238 INFO net.sf.ehcache.pool.sizeof.AgentLoader: Trying to load agent @ /tmp/ehcache-sizeof-agent5050905318542369702.jar
2012-12-17 01:59:31,244 INFO net.sf.ehcache.pool.impl.DefaultSizeOfEngine: using Agent sizeof engine
2012-12-17 01:59:31,273 INFO net.sf.ehcache.pool.impl.DefaultSizeOfEngine: using Agent sizeof engine
2012-12-17 02:26:47,938 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 1 heartbeats
2012-12-17 02:26:52,939 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 2 heartbeats
2012-12-17 02:26:57,941 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 3 heartbeats
2012-12-17 02:27:02,943 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 4 heartbeats
2012-12-17 02:27:07,942 ERROR com.cloudera.flume.agent.LogicalNode: Forcing driver to exit uncleanly
2012-12-17 02:27:07,943 ERROR com.cloudera.flume.core.connector.DirectDriver: Closing down due to exception during append calls
2012-12-17 02:27:07,943 INFO com.cloudera.flume.agent.LogicalNode: Node config successfully set to FlumeConfigData: {srcVer:'Mon Dec 17 02:26:37 EST 2012' snkVer:'Mon Dec 17 02:26:37 EST 2012'  ts='Mon Dec 17 02:26:37 EST 2012' flowId:'default-flow' source:'tail( "/var/log/flume/test1.log", true )' sink:'{ value( "LogType", "Test1Log" ) => agentBESink( "ip-10-40-222-77.ec2.internal", 35853 ) }' }
2012-12-17 02:27:07,944 INFO com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode ec2-75-101-165-36.compute-1.amazonaws.com-19 exited with error: Waiting for queue element was interrupted! null
java.io.IOException: Waiting for queue element was interrupted! null
        at com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:222)
        at com.cloudera.flume.collector.CollectorSource.next(CollectorSource.java:72)
        at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:108)
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2038)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
        at com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:209)
        ... 2 more
2012-12-17 02:27:07,944 INFO com.cloudera.flume.collector.CollectorSource: closed
2012-12-17 02:27:07,946 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Closed server on port 35853...
2012-12-17 02:27:07,948 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Queue still has 0 elements ...
2012-12-17 02:27:07,949 ERROR com.cloudera.flume.core.connector.DirectDriver: Exiting driver logicalNode ec2-75-101-165-36.compute-1.amazonaws.com-19 in error state CollectorSource | EHCacheSink because Waiting for queue element was interrupted! null
2012-12-17 02:27:07,952 INFO com.cloudera.flume.handlers.debug.InsistentOpenDecorator: open attempt 0 failed, backoff (1000ms): Failed to open thrift event sink to ip-10-40-222-77.ec2.internal:35853 : java.net.ConnectException: Connection refused
2012-12-17 02:27:08,953 INFO com.cloudera.flume.handlers.debug.InsistentOpenDecorator: open attempt 1 failed, backoff (2000ms): Failed to open thrift event sink to ip-10-40-222-77.ec2.internal:35853 : java.net.ConnectException: Connection refused
2012-12-17 02:27:10,953 INFO com.cloudera.flume.handlers.debug.I