Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Problems with Flume on windows with HTTP source

Copy link to this message
Re: Problems with Flume on windows with HTTP source
Nickolay Kolev 2013-06-21, 06:34
Hi Hari and the all members of this list,

Thanks for pointing me that error. This is actually my second attempt to
use Apache Flume - with avro client. The symptoms are the same - the is no
listener on the specified port and I am not able to connect.

I started to evaluate Flume just yesterday and many things are new to me
and probably I do many mistakes. I want to point that I use Flume 1.3.1
compiled by me if this is important to be mentioned. I have tested too with
the official binary from Flume web site but the results are the same.

Here is what I do to test HTTP source:

my config file (httppost.conf):
agent1.sources = r1
agent1.channels = ch1
agent1.sinks = k1

agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
agent1.sources.r1.port = 9001
agent1.sources.r1.channels = ch1
#agent1.sources.r1.handler = org.example.rest.RestHandler
agent1.sources.r1.handler = org.apache.flume.http.JSONHandler
agent1.sources.r1.handler.nickname = json handler

agent1.sinks.k1.type = file_roll
agent1.sinks.k1.channel = ch1
agent1.sinks.k1.sink.directory C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log

agent1.channels.ch1.type = memory
agent1.channels.ch1.capacity = 10000
agent1.channels.ch1.transactionCapacity = 1000

my server bat file:
set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
set JAVA_OPTS=-Xmx60m
set CONF=%FLUME_HOME%\conf\httppost.conf
set AGENT=agent1

-Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
"%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%

How I test:

server console:
>"C:\Java\jdk1.6.0_45\bin\java.exe" -Xmx60m
-cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\\lib\*"
org.apache.flume.node.Application -f
C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\httppost.conf -n agent1

client console:
>C:\Projects\nickyk\apache-flume-1.3.1-bin>curl -X POST -H "Content-Type:
application/json" -d "[{ \"headers\" : { \"timestamp\" : \"434324343\",
\"host\" : \"random_host.example.co
m\" }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \"
namenode.example.com\", \"datanode\" : \"random_datanode.example.com\" },
\"body\" : \"really_random_body\"
}]" http://localhost:9001
curl: (7) Failed connect to localhost:9001; No error

If I understand correct the documentation I should expect that on the
specified port (9001) it must be listener that will accept POST requests?
Is this correct?
Do I miss some important aspects that are critical to run Flume?

Thanks in advance for your help.
best regards,
nickolay kolev

On Thu, Jun 20, 2013 at 9:07 PM, Hari Shreedharan <[EMAIL PROTECTED]
> wrote:

>  Does not look like you are using the http source at all. Your source type
> needs to be HTTP
> Cheers,
> Hari
> On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote:
> Hi all,
> I am new to flume and all that logging stuff and probably many things are
> unclear to me despite I read the docs.
> I want to run Flume on windows server to create proof of concept solution.
> I was able to compile it successfully. I was able to make successful test
> with netcat.
> Next test that i wanted to do is with http source but unfortunately it did
> not pass.
>  Here is what I do:
> my server config (httppost.conf):
> agent1.sources = r1
> agent1.channels = memoryChannel
> agent1.sinks = k1
> agent1.sources.r1.type = avro
> agent1.sources.r1.bind = localhost
> agent1.sources.r1.port = 9003
> agent1.sinks.k1.type = file_roll
> agent1.sinks.k1.sink.directory > C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
> agent1.channels.memoryChannel.type = memory
> agent1.channels.memoryChannel.capacity = 10000
> agent1.channels.memoryChannel.transactionCapacity = 1000
> agent1.sources.r1.channels = memoryChannel
> agent1.sinks.k1.channel = memoryChannel
> I run the server with this bat file:
> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\