Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Problems with Flume on windows with HTTP source


Copy link to this message
-
Problems with Flume on windows with HTTP source
Hi all,
I am new to flume and all that logging stuff and probably many things are
unclear to me despite I read the docs.
I want to run Flume on windows server to create proof of concept solution.
I was able to compile it successfully. I was able to make successful test
with netcat.
Next test that i wanted to do is with http source but unfortunately it did
not pass.
 Here is what I do:

my server config (httppost.conf):
agent1.sources = r1
agent1.channels = memoryChannel
agent1.sinks = k1

agent1.sources.r1.type = avro
agent1.sources.r1.bind = localhost
agent1.sources.r1.port = 9003

agent1.sinks.k1.type = file_roll
agent1.sinks.k1.sink.directory C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log

agent1.channels.memoryChannel.type = memory
agent1.channels.memoryChannel.capacity = 10000
agent1.channels.memoryChannel.transactionCapacity = 1000
agent1.sources.r1.channels = memoryChannel
agent1.sinks.k1.channel = memoryChannel

I run the server with this bat file:
set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
set JAVA_OPTS=-Xmx60m
set CONF=%FLUME_HOME%\conf\httppost.conf
set AGENT=agent1

%JAVA% %JAVA_OPTS%
-Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
"%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%

netstat does not show any listener on port 9003? Is this normal or there
must be a listener?

I run avro client with this bat file:
set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin
set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
set JAVA_OPTS=-Xmx60m
set AVRO_HOST=localhost
set AVRO_PORT=9003
set SENDFILE=%FLUME_HOME%\logs\data.log
set HEADFILE=%FLUME_HOME%\conf\header.txt

%JAVA%  %JAVA_OPTS%
-Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
"%FLUME_HOME%\lib\*" org.apache.flume.client.avro.AvroCLIClient -H
%AVRO_HOST% -p %AVRO_PORT%  -F %SENDFILE%  --headerFile %HEADFILE%
Then I get this output:
C:\Projects\nickyk\apache-flume-1.3.1-bin>"C:\Java\jdk1.6.0_45\bin\java.exe"
-Xmx60m
-Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\log4j.properties
-cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\lib\*"
org.apache.flume.client.avro.AvroCLIClient -H  localhost -p 9003  -F
C:\Projects\nickyk\apache-flume-1.3.1-bin\logs\data.log --headerFile
C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\header.txt
2013-06-20 17:55:58,285 (main) [DEBUG -
org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)]
Inserting Header Key [filename] header value [datalog]
2013-06-20 17:55:58,287 (main) [DEBUG -
org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)]
Inserting Header Key [host] header value [zhouhh]
2013-06-20 17:55:58,295 (main) [DEBUG -
org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)]
Batch size string = 5
2013-06-20 17:55:59,358 (main) [ERROR -
org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:79)]
Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost,
port: 9003 }: RPC connection error
        at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
        at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
        at
org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
        at
org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
        at
org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
        at
org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:9003
        at
org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
        at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
        at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
        at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
        ... 5 more
Caused by: java.net.ConnectException: Connection refused: no further
information
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
        at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
        at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
        at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
        at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
2013-06-20 17:55:59,369 (main) [DEBUG -
org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:84)]
Exiting
My question is what I am doing wrong and what I need to test in order to
fix this situatioin.

Thanks in advance.

best regards,
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB