Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Problems with Flume on windows with HTTP source


+
Nickolay Kolev 2013-06-20, 15:57
+
Hari Shreedharan 2013-06-20, 18:07
Copy link to this message
-
Re: Problems with Flume on windows with HTTP source
Hi Hari and the all members of this list,

Thanks for pointing me that error. This is actually my second attempt to
use Apache Flume - with avro client. The symptoms are the same - the is no
listener on the specified port and I am not able to connect.

I started to evaluate Flume just yesterday and many things are new to me
and probably I do many mistakes. I want to point that I use Flume 1.3.1
compiled by me if this is important to be mentioned. I have tested too with
the official binary from Flume web site but the results are the same.

Here is what I do to test HTTP source:

my config file (httppost.conf):
agent1.sources = r1
agent1.channels = ch1
agent1.sinks = k1

agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
agent1.sources.r1.port = 9001
agent1.sources.r1.channels = ch1
#agent1.sources.r1.handler = org.example.rest.RestHandler
agent1.sources.r1.handler = org.apache.flume.http.JSONHandler
agent1.sources.r1.handler.nickname = json handler

agent1.sinks.k1.type = file_roll
agent1.sinks.k1.channel = ch1
agent1.sinks.k1.sink.directory C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log

agent1.channels.ch1.type = memory
agent1.channels.ch1.capacity = 10000
agent1.channels.ch1.transactionCapacity = 1000

my server bat file:
set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
set JAVA_OPTS=-Xmx60m
set CONF=%FLUME_HOME%\conf\httppost.conf
set AGENT=agent1

%JAVA% %JAVA_OPTS%
-Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
"%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%

How I test:

server console:
>"C:\Java\jdk1.6.0_45\bin\java.exe" -Xmx60m
-Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\log4j.properties
-cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\\lib\*"
org.apache.flume.node.Application -f
C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\httppost.conf -n agent1

client console:
>C:\Projects\nickyk\apache-flume-1.3.1-bin>curl -X POST -H "Content-Type:
application/json" -d "[{ \"headers\" : { \"timestamp\" : \"434324343\",
\"host\" : \"random_host.example.co
m\" }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \"
namenode.example.com\", \"datanode\" : \"random_datanode.example.com\" },
\"body\" : \"really_random_body\"
}]" http://localhost:9001
curl: (7) Failed connect to localhost:9001; No error

If I understand correct the documentation I should expect that on the
specified port (9001) it must be listener that will accept POST requests?
Is this correct?
Do I miss some important aspects that are critical to run Flume?

Thanks in advance for your help.
best regards,
nickolay kolev

On Thu, Jun 20, 2013 at 9:07 PM, Hari Shreedharan <[EMAIL PROTECTED]
> wrote:

>  Does not look like you are using the http source at all. Your source type
> needs to be HTTP
>
>
> Cheers,
> Hari
>
> On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote:
>
> Hi all,
> I am new to flume and all that logging stuff and probably many things are
> unclear to me despite I read the docs.
> I want to run Flume on windows server to create proof of concept solution.
> I was able to compile it successfully. I was able to make successful test
> with netcat.
> Next test that i wanted to do is with http source but unfortunately it did
> not pass.
>  Here is what I do:
>
> my server config (httppost.conf):
> agent1.sources = r1
> agent1.channels = memoryChannel
> agent1.sinks = k1
>
> agent1.sources.r1.type = avro
> agent1.sources.r1.bind = localhost
> agent1.sources.r1.port = 9003
>
> agent1.sinks.k1.type = file_roll
> agent1.sinks.k1.sink.directory > C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
>
> agent1.channels.memoryChannel.type = memory
> agent1.channels.memoryChannel.capacity = 10000
> agent1.channels.memoryChannel.transactionCapacity = 1000
>
>
> agent1.sources.r1.channels = memoryChannel
> agent1.sinks.k1.channel = memoryChannel
>
> I run the server with this bat file:
> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
+
Ashish 2013-06-21, 06:48
+
Nickolay Kolev 2013-06-21, 07:03
+
Ashish 2013-06-21, 07:40
+
Nickolay Kolev 2013-06-21, 08:29
+
Nickolay Kolev 2013-06-21, 08:20
+
Ashish 2013-06-21, 08:25
+
Nickolay Kolev 2013-06-21, 07:52
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB