Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Need you to check if Flume source/sink configuration is ok?


Copy link to this message
-
FW: Need you to check if Flume source/sink configuration is ok?
To add to the below information,

My motive is to start a flow where a single node acts as a flume agent and sends data to a collector(which is the same node). Is it possible? If yes, how do  I set it up! Please help.
I am in a real fix now!

Thanks and Regards,
Shouvanik Haldar | Cloud SME Pool | Mobile:+91-9830017568

From: Haldar, Shouvanik
Sent: Monday, December 17, 2012 1:15 PM
To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
Subject: RE: Need you to check if Flume source/sink configuration is ok?

As advised, I tried to configure the same node to send data to the active collector. I did the following changes via Flume Master Web console:

[cid:[EMAIL PROTECTED]2FE0]
When I press "Submit Query", and I check the Flume logs ->

okeeper/zookeeper.jar:/usr/lib/flume/build/classes
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/flume/lib::/usr/lib/hadoop/lib/native/Linux-amd64-64
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-131.17.1.el6.x86_64
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=flume
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/var/run/flume
2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/usr/lib/flume
2012-12-17 01:59:28,278 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
2012-12-17 01:59:28,315 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181
2012-12-17 01:59:28,333 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
2012-12-17 01:59:28,351 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13b82b043240027, negotiated timeout = 40000
2012-12-17 01:59:30,673 INFO org.terracotta.modules.ehcache.store.ClusteredStore: Cache [RTLogCache] using concurrency: 256
2012-12-17 01:59:30,896 INFO net.sf.ehcache.pool.sizeof.JvmInformation: Detected JVM data model settings of: 64-Bit HotSpot JVM with Compressed OOPs
2012-12-17 01:59:31,238 INFO net.sf.ehcache.pool.sizeof.AgentLoader: Extracted agent jar to temporary file /tmp/ehcache-sizeof-agent5050905318542369702.jar
2012-12-17 01:59:31,238 INFO net.sf.ehcache.pool.sizeof.AgentLoader: Trying to load agent @ /tmp/ehcache-sizeof-agent5050905318542369702.jar
2012-12-17 01:59:31,244 INFO net.sf.ehcache.pool.impl.DefaultSizeOfEngine: using Agent sizeof engine
2012-12-17 01:59:31,273 INFO net.sf.ehcache.pool.impl.DefaultSizeOfEngine: using Agent sizeof engine
2012-12-17 02:26:47,938 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 1 heartbeats
2012-12-17 02:26:52,939 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 2 heartbeats
2012-12-17 02:26:57,941 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 3 heartbeats
2012-12-17 02:27:02,943 WARN com.cloudera.flume.agent.LivenessManager: Heartbeats are backing up, currently behind by 4 heartbeats
2012-12-17 02:27:07,942 ERROR com.cloudera.flume.agent.LogicalNode: Forcing driver to exit uncleanly
2012-12-17 02:27:07,943 ERROR com.cloudera.flume.core.connector.DirectDriver: Closing down due to exception during append calls
2012-12-17 02:27:07,943 INFO com.cloudera.flume.agent.LogicalNode: Node config successfully set to FlumeConfigData: {srcVer:'Mon Dec 17 02:26:37 EST 2012' snkVer:'Mon Dec 17 02:26:37 EST 2012'  ts='Mon Dec 17 02:26:37 EST 2012' flowId:'default-flow' source:'tail( "/var/log/flume/test1.log", true )' sink:'{ value( "LogType", "Test1Log" ) => agentBESink( "ip-10-40-222-77.ec2.internal", 35853 ) }' }
2012-12-17 02:27:07,944 INFO com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode ec2-75-101-165-36.compute-1.amazonaws.com-19 exited with error: Waiting for queue element was interrupted! null
java.io.IOException: Waiting for queue element was interrupted! null
        at com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:222)
        at com.cloudera.flume.collector.CollectorSource.next(CollectorSource.java:72)
        at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:108)
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2038)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
        at com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:209)
        ... 2 more
2012-12-17 02:27:07,944 INFO com.cloudera.flume.collector.CollectorSource: closed
2012-12-17 02:27:07,946 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Closed server on port 35853...
2012-12-17 02:27:07,948 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Queue still has 0 elements ...
2012-12-17 02:27:07,949 ERROR com.cloudera.flume.core.connector.DirectDriver: Exiting driver logicalNode ec2-75-101-165-36.compute-1.amazonaws.com-19 in error state CollectorSource | EHCacheSink because Waiting for queue element was interrupted! null
2012-12-17 02:27:07,952 INFO com.cloudera.flume.h