Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Problems with Flume on windows with HTTP source


Copy link to this message
-
Re: Problems with Flume on windows with HTTP source
Does not look like you are using the http source at all. Your source type needs to be HTTP
Cheers,
Hari
On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote:

> Hi all,
> I am new to flume and all that logging stuff and probably many things are unclear to me despite I read the docs.
> I want to run Flume on windows server to create proof of concept solution. I was able to compile it successfully. I was able to make successful test with netcat.
> Next test that i wanted to do is with http source but unfortunately it did not pass.
>  Here is what I do:
>
> my server config (httppost.conf):
> agent1.sources = r1
> agent1.channels = memoryChannel
> agent1.sinks = k1
>
> agent1.sources.r1.type = avro
> agent1.sources.r1.bind = localhost
> agent1.sources.r1.port = 9003
>
> agent1.sinks.k1.type = file_roll
> agent1.sinks.k1.sink.directory = C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
>
> agent1.channels.memoryChannel.type = memory
> agent1.channels.memoryChannel.capacity = 10000
> agent1.channels.memoryChannel.transactionCapacity = 1000
>
>
> agent1.sources.r1.channels = memoryChannel
> agent1.sinks.k1.channel = memoryChannel
>  
> I run the server with this bat file:
> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
> set JAVA_OPTS=-Xmx60m
> set CONF=%FLUME_HOME%\conf\httppost.conf
> set AGENT=agent1
>
> %JAVA% %JAVA_OPTS%  -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%
>
> netstat does not show any listener on port 9003? Is this normal or there must be a listener?
>
> I run avro client with this bat file:
> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin
> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
> set JAVA_OPTS=-Xmx60m
> set AVRO_HOST=localhost
> set AVRO_PORT=9003
> set SENDFILE=%FLUME_HOME%\logs\data.log
> set HEADFILE=%FLUME_HOME%\conf\header.txt
>
> %JAVA%  %JAVA_OPTS%  -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp "%FLUME_HOME%\lib\*" org.apache.flume.client.avro.AvroCLIClient -H  %AVRO_HOST% -p %AVRO_PORT%  -F %SENDFILE%  --headerFile %HEADFILE%
>
>
> Then I get this output:
> C:\Projects\nickyk\apache-flume-1.3.1-bin>"C:\Java\jdk1.6.0_45\bin\java.exe"  -Xmx60m  -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\log4j.properties  -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\lib\*" org.apache.flume.client.avro.AvroCLIClient -H  localhost -p 9003  -F C:\Projects\nickyk\apache-flume-1.3.1-bin\logs\data.log --headerFile C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\header.txt
> 2013-06-20 17:55:58,285 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] Inserting Header Key [filename] header value [datalog]
> 2013-06-20 17:55:58,287 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] Inserting Header Key [host] header value [zhouhh]
> 2013-06-20 17:55:58,295 (main) [DEBUG - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] Batch size string = 5
> 2013-06-20 17:55:59,358 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:79)] Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 9003 }: RPC connection error
>         at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>         at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>         at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>         at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>         at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>         at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:9003 (http://127.0.0.1:9003)