Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Problems with Flume on windows with HTTP source


Copy link to this message
-
Re: Problems with Flume on windows with HTTP source
Ashish 2013-06-21, 07:40
I am using Flume 1.3.1 and ran the example before sending.

I don't have access to a windows system at the moment so can't test.

This is output at my end on Flume Agent console

2013-06-21 13:03:54,913 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
Starting Source src1
2013-06-21 13:03:54,927 (lifecycleSupervisor-1-2) [INFO -
org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2013-06-21 13:03:54,958 (lifecycleSupervisor-1-2) [INFO -
org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] jetty-6.1.26
2013-06-21 13:03:55,022 (lifecycleSupervisor-1-2) [INFO -
org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Started
SocketConnector@0.0.0.0:9001
2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)]
Event: { headers:{timestamp=434324343, host=random_host.example.co
m} body: 72 61 6E 64 6F 6D 5F 62 6F 64 79                random_body }
2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)]
Event: { headers:{namenode=namenode.example.com, datanoderandom_datanode.example.com} body: 72 65 61 6C 6C 79 5F 72 61 6E 64 6F 6D
5F 62 6F really_random_bo }

Flume dev's can comment further on this.
On Fri, Jun 21, 2013 at 12:33 PM, Nickolay Kolev <[EMAIL PROTECTED]> wrote:

> Hi Ashish,
>
> Thanks for pointing me that error. I am trying to read the code and this
> is the correct full class name. (last time I wrote java code was in 1998
> and my knowledge are a lot out of date)
>
> Unfortunately the result is the same.
> >curl -X POST -H "Content-Type: application/json" -d "[{ \"headers\" : {
> \"timestamp\" : \"434324343\", \"host\" : \"random_host.example.com\" },
> \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \"
> namenode.example.com\", \"datanode\" : \"random_datanode.example.com\" },
> \"body\" : \"really_random_body\"}]" http://localhost:9001
>
> curl: (7) Failed connect to localhost:9001; No error
>
> Could be the reason if this errors the fact that I use Windows? I see in
> wiki that Windows is not officially supported.
>
> Yesterday my first test was with netcat source and it worked as expected.
>
>
> best regards,
> nickolay kolev
>
>
> On Fri, Jun 21, 2013 at 9:48 AM, Ashish <[EMAIL PROTECTED]> wrote:
>
>> update the handler config as
>>
>> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
>>
>> and try. I used your source config and it worked for me. Let us know if
>> it work
>>
>>
>> On Fri, Jun 21, 2013 at 12:04 PM, Nickolay Kolev <[EMAIL PROTECTED]>wrote:
>>
>>> Hi Hari and the all members of this list,
>>>
>>> Thanks for pointing me that error. This is actually my second attempt to
>>> use Apache Flume - with avro client. The symptoms are the same - the is no
>>> listener on the specified port and I am not able to connect.
>>>
>>> I started to evaluate Flume just yesterday and many things are new to me
>>> and probably I do many mistakes. I want to point that I use Flume 1.3.1
>>> compiled by me if this is important to be mentioned. I have tested too with
>>> the official binary from Flume web site but the results are the same.
>>>
>>> Here is what I do to test HTTP source:
>>>
>>> my config file (httppost.conf):
>>> agent1.sources = r1
>>> agent1.channels = ch1
>>> agent1.sinks = k1
>>>
>>> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
>>> agent1.sources.r1.port = 9001
>>> agent1.sources.r1.channels = ch1
>>> #agent1.sources.r1.handler = org.example.rest.RestHandler
>>> agent1.sources.r1.handler = org.apache.flume.http.JSONHandler
>>> agent1.sources.r1.handler.nickname = json handler
>>>
>>> agent1.sinks.k1.type = file_roll
>>> agent1.sinks.k1.channel = ch1
>>>
>>> agent1.sinks.k1.sink.directory >>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal