Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - seeking help on flume cluster deployment


Copy link to this message
-
Re: seeking help on flume cluster deployment
Chen Wang 2014-01-11, 06:09
Never mind.
Stupid me. I load config from property files and there is extra space there.
Thanks,
Chen
On Fri, Jan 10, 2014 at 5:47 PM, Chen Wang <[EMAIL PROTECTED]>wrote:

> I tried to telnet, also get connection refused:
>  telnet localhost 4141
> Trying ::1...
> telnet: connect to address ::1: Connection refused
> Trying 127.0.0.1...
> telnet: connect to address 127.0.0.1: Connection refused
>
>
> On Fri, Jan 10, 2014 at 5:15 PM, Chen Wang <[EMAIL PROTECTED]>wrote:
>
>> Hey guys,
>> I think i still need some help on the custom flume client. Right now I
>> have finished the Avro sink client in my storm bolt. On a test machine, i
>> started a flume agent:
>> StormAgent.sources = avro
>> StormAgent.channels = MemChannel
>> StormAgent.sinks = HDFS
>>
>> StormAgent.sources.avro.type = avro
>> StormAgent.sources.avro.channels = MemChannel
>> StormAgent.sources.avro.bind = localhost
>> StormAgent.sources.avro.port = 10001
>>
>> I assume this will automatically wait on the localhost:10001?
>>
>> When I run my LoadBalancingRpcClient. on the same machine, I receive
>> connection refused exception:
>> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost,
>> port: 10001 }: RPC connection error
>> at
>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:161)
>>  at
>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:115)
>> at
>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:590)
>>  at
>> org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88)
>> at
>> org.apache.flume.api.LoadBalancingRpcClient.createClient(LoadBalancingRpcClient.java:214)
>>  at
>> org.apache.flume.api.LoadBalancingRpcClient.getClient(LoadBalancingRpcClient.java:197)
>> at
>> org.apache.flume.api.LoadBalancingRpcClient.append(LoadBalancingRpcClient.java:71)
>>  at
>> com.walmartlabs.targeting.storm.bolt.HubbleStreamAvroSinkBolt.execute(HubbleStreamAvroSinkBolt.java:89)
>> at
>> backtype.storm.daemon.executor$fn__4050$tuple_action_fn__4052.invoke(executor.clj:566)
>>  at
>> backtype.storm.daemon.executor$mk_task_receiver$fn__3976.invoke(executor.clj:345)
>> at
>> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
>>  at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
>>  at
>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
>> at
>> backtype.storm.daemon.executor$fn__4050$fn__4059$fn__4106.invoke(executor.clj:658)
>>  at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
>> at clojure.lang.AFn.run(AFn.java:24)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.IOException: Error connecting to localhost/
>> 127.0.0.1:10001
>>
>> Is this still some config issue? I tried ip address as well, but with the
>> same error. I am this close now...
>> Thank you for any help!
>> Chen
>>
>>
>>
>> On Thu, Jan 9, 2014 at 10:09 PM, Chen Wang <[EMAIL PROTECTED]>wrote:
>>
>>> Ashish,
>>> Interesting enough, i was initially doing 1 too, and had a working
>>> version. But finally I give it up because in my bolt i have to flush to
>>> hdfs either when data reaching certain size or a timer times out, which is
>>> exactly what flume can offer. Also it has some complexity of grouping
>>> entries within the same partition while with flume it is a piece of cake.
>>>
>>> Thank you so much for all you guys's input. It helped me a lot !
>>> Chen
>>>
>>>
>>>
>>> On Thu, Jan 9, 2014 at 10:00 PM, Ashish <[EMAIL PROTECTED]> wrote:
>>>
>>>> Got it!
>>>>
>>>> My first reaction was to use HDFS bolt to write data directly to HDFS,
>>>> but couldn't find an implementation for the same. My knowledge is limited
>>>> for Storm.
>>>> If the data is already flowing through Storm, you got two options
>>>> 1. Write a bolt to dump data to HDFS
>>>> 2. Write a Flume bolt using RPC client as recommended in thread, and