Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> seeking help on flume cluster deployment


Copy link to this message
-
Re: seeking help on flume cluster deployment
Never mind.
Stupid me. I load config from property files and there is extra space there.
Thanks,
Chen
On Fri, Jan 10, 2014 at 5:47 PM, Chen Wang <[EMAIL PROTECTED]>wrote:

> I tried to telnet, also get connection refused:
>  telnet localhost 4141
> Trying ::1...
> telnet: connect to address ::1: Connection refused
> Trying 127.0.0.1...
> telnet: connect to address 127.0.0.1: Connection refused
>
>
> On Fri, Jan 10, 2014 at 5:15 PM, Chen Wang <[EMAIL PROTECTED]>wrote:
>
>> Hey guys,
>> I think i still need some help on the custom flume client. Right now I
>> have finished the Avro sink client in my storm bolt. On a test machine, i
>> started a flume agent:
>> StormAgent.sources = avro
>> StormAgent.channels = MemChannel
>> StormAgent.sinks = HDFS
>>
>> StormAgent.sources.avro.type = avro
>> StormAgent.sources.avro.channels = MemChannel
>> StormAgent.sources.avro.bind = localhost
>> StormAgent.sources.avro.port = 10001
>>
>> I assume this will automatically wait on the localhost:10001?
>>
>> When I run my LoadBalancingRpcClient. on the same machine, I receive
>> connection refused exception:
>> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost,
>> port: 10001 }: RPC connection error
>> at
>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:161)
>>  at
>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:115)
>> at
>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:590)
>>  at
>> org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88)
>> at
>> org.apache.flume.api.LoadBalancingRpcClient.createClient(LoadBalancingRpcClient.java:214)
>>  at
>> org.apache.flume.api.LoadBalancingRpcClient.getClient(LoadBalancingRpcClient.java:197)
>> at
>> org.apache.flume.api.LoadBalancingRpcClient.append(LoadBalancingRpcClient.java:71)
>>  at
>> com.walmartlabs.targeting.storm.bolt.HubbleStreamAvroSinkBolt.execute(HubbleStreamAvroSinkBolt.java:89)
>> at
>> backtype.storm.daemon.executor$fn__4050$tuple_action_fn__4052.invoke(executor.clj:566)
>>  at
>> backtype.storm.daemon.executor$mk_task_receiver$fn__3976.invoke(executor.clj:345)
>> at
>> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
>>  at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
>>  at
>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
>> at
>> backtype.storm.daemon.executor$fn__4050$fn__4059$fn__4106.invoke(executor.clj:658)
>>  at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
>> at clojure.lang.AFn.run(AFn.java:24)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.IOException: Error connecting to localhost/
>> 127.0.0.1:10001
>>
>> Is this still some config issue? I tried ip address as well, but with the
>> same error. I am this close now...
>> Thank you for any help!
>> Chen
>>
>>
>>
>> On Thu, Jan 9, 2014 at 10:09 PM, Chen Wang <[EMAIL PROTECTED]>wrote:
>>
>>> Ashish,
>>> Interesting enough, i was initially doing 1 too, and had a working
>>> version. But finally I give it up because in my bolt i have to flush to
>>> hdfs either when data reaching certain size or a timer times out, which is
>>> exactly what flume can offer. Also it has some complexity of grouping
>>> entries within the same partition while with flume it is a piece of cake.
>>>
>>> Thank you so much for all you guys's input. It helped me a lot !
>>> Chen
>>>
>>>
>>>
>>> On Thu, Jan 9, 2014 at 10:00 PM, Ashish <[EMAIL PROTECTED]> wrote:
>>>
>>>> Got it!
>>>>
>>>> My first reaction was to use HDFS bolt to write data directly to HDFS,
>>>> but couldn't find an implementation for the same. My knowledge is limited
>>>> for Storm.
>>>> If the data is already flowing through Storm, you got two options
>>>> 1. Write a bolt to dump data to HDFS
>>>> 2. Write a Flume bolt using RPC client as recommended in thread, and
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB