Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Problems with Flume on windows with HTTP source


+
Nickolay Kolev 2013-06-20, 15:57
Copy link to this message
-
Re: Problems with Flume on windows with HTTP source
Hari Shreedharan 2013-06-20, 18:07
Does not look like you are using the http source at all. Your source type needs to be HTTP
Cheers,
Hari
On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote:

> Hi all,
> I am new to flume and all that logging stuff and probably many things are unclear to me despite I read the docs.
> I want to run Flume on windows server to create proof of concept solution. I was able to compile it successfully. I was able to make successful test with netcat.
> Next test that i wanted to do is with http source but unfortunately it did not pass.
>  Here is what I do:
>
> my server config (httppost.conf):
> agent1.sources = r1
> agent1.channels = memoryChannel
> agent1.sinks = k1
>
> agent1.sources.r1.type = avro
> agent1.sources.r1.bind = localhost
> agent1.sources.r1.port = 9003
>
> agent1.sinks.k1.type = file_roll
> agent1.sinks.k1.sink.directory = C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
>
> agent1.channels.memoryChannel.type = memory
> agent1.channels.memoryChannel.capacity = 10000
> agent1.channels.memoryChannel.transactionCapacity = 1000
>
>
> agent1.sources.r1.channels = memoryChannel
> agent1.sinks.k1.channel = memoryChannel
>  
> I run the server with this bat file:
> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
> set JAVA_OPTS=-Xmx60m
> set CONF=%FLUME_HOME%\conf\httppost.conf
> set AGENT=agent1
>
> %JAVA% %JAVA_OPTS%  -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%
>
> netstat does not show any listener on port 9003? Is this normal or there must be a listener?
>
> I run avro client with this bat file:
> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin
> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
> set JAVA_OPTS=-Xmx60m
> set AVRO_HOST=localhost
> set AVRO_PORT=9003
> set SENDFILE=%FLUME_HOME%\logs\data.log
> set HEADFILE=%FLUME_HOME%\conf\header.txt
>
> %JAVA%  %JAVA_OPTS%  -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp "%FLUME_HOME%\lib\*" org.apache.flume.client.avro.AvroCLIClient -H  %AVRO_HOST% -p %AVRO_PORT%  -F %SENDFILE%  --headerFile %HEADFILE%
>
>
> Then I get this output:
> C:\Projects\nickyk\apache-flume-1.3.1-bin>"C:\Java\jdk1.6.0_45\bin\java.exe"  -Xmx60m  -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\log4j.properties  -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\lib\*" org.apache.flume.client.avro.AvroCLIClient -H  localhost -p 9003  -F C:\Projects\nickyk\apache-flume-1.3.1-bin\logs\data.log --headerFile C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\header.txt
> 2013-06-20 17:55:58,285 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] Inserting Header Key [filename] header value [datalog]
> 2013-06-20 17:55:58,287 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] Inserting Header Key [host] header value [zhouhh]
> 2013-06-20 17:55:58,295 (main) [DEBUG - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] Batch size string = 5
> 2013-06-20 17:55:59,358 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:79)] Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 9003 }: RPC connection error
>         at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>         at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>         at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>         at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>         at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>         at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:9003 (http://127.0.0.1:9003)
+
Nickolay Kolev 2013-06-21, 06:34
+
Ashish 2013-06-21, 06:48
+
Nickolay Kolev 2013-06-21, 07:03
+
Ashish 2013-06-21, 07:40
+
Nickolay Kolev 2013-06-21, 08:29
+
Nickolay Kolev 2013-06-21, 08:20
+
Ashish 2013-06-21, 08:25
+
Nickolay Kolev 2013-06-21, 07:52