Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Need you to check if Flume source/sink configuration is ok?


Copy link to this message
-
Re: FW: Need you to check if Flume source/sink configuration is ok?
Yes, it is possible.
You need another nod name for it (another configuration in different
name); actually, you used primary logical node which has same name with
physical node but you better go with two node names; one for agent role,
the other for collector role; just let primary logical node "IDLE" by
unmap-ing any configuration from it.

For example of two logical nodes on a physical node, in configuration
tool, you should use "or specify another node" input box to specify
another node name.
say "agent1" for agent role, and "collector1" for collector role, you
should specify them using that input box. And put proper source and sink
respectively.

Once you register configurations by specifying node names; you will see
those in configuration list; not it status as you don't map them yet.
The next step is to map those to a physical node which is
es2...36.compute-1.amazonaws.com; I recommend to map collector first and
then agent.

now, hopefully you will see those nodes in Status list in master web UI;
check they are all ACTIVE and then append some logs to log file to see
if you get data delivered to target storage.

For your understanding of what you did,
as in configuration, you used running node name
(es2...36.compute-1.amazonaws.com), which is the primary logical node,
your configuration was directly sent down to the node and applied by
replacing existing configuration; meanwhile, there was interruption
error because there was no log data inflow I guess; when node stops,
node waits for about 30 secs for data inflow and then interruption
happens to force it to stop.
So you ended up with just having replaced configurations for the same
logical node rather than starting another new logical node for another
role and connecting them as a flow.

Hope it is helpful.

JS

On 12/17/12 4:53 PM, [EMAIL PROTECTED] wrote:
>
> To add to the below information,
>
> My motive is to start a flow where a single node acts as a flume agent
> and sends data to a collector(which is the same node). Is it possible?
> If yes, how do  I set it up! Please help.
>
> I am in a real fix now!
>
> *Thanks and Regards,*
>
> *Shouvanik Haldar | Cloud SME Pool | Mobile:+91-9830017568 *
>
> *From:*Haldar, Shouvanik
> *Sent:* Monday, December 17, 2012 1:15 PM
> *To:* '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
> *Subject:* RE: Need you to check if Flume source/sink configuration is ok?
>
> As advised, I tried to configure the same node to send data to the
> active collector. I did the following changes via Flume Master Web
> console:
>
> **
>
> When I press "Submit Query", and I check the Flume logs ->
>
> okeeper/zookeeper.jar:/usr/lib/flume/build/classes
>
> 2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.library.path=/usr/lib/flume/lib::/usr/lib/hadoop/lib/native/Linux-amd64-64
>
> 2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
>
> 2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
>
> 2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.name=Linux
>
> 2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
>
> 2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.version=2.6.32-131.17.1.el6.x86_64
>
> 2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.name=flume
>
> 2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.home=/var/run/flume
>
> 2012-12-17 01:59:28,277 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.dir=/usr/lib/flume
>
> 2012-12-17 01:59:28,278 INFO org.apache.zookeeper.ZooKeeper:
> Initiating client connection, connectString=localhost:2181
> sessionTimeout=180000 watcher=hconnection
>
> 2012-12-17 01:59:28,315 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server localhost/127.0.0.1:2181
Jeong-shik Jang / [EMAIL PROTECTED]
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting