Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - avro client fails to connect to flume
Siva Gudavalli
2012-11-01, 15:13
Roshan Naik
2012-11-01, 22:39
Siva Gudavalli
2012-11-02, 01:44
Siva Gudavalli
2012-11-02, 04:16
Alexander Lorenz
2012-11-07, 12:58
Copy link to this message
-
Re: avro client fails to connect to flume
Hari Shreedharan 2012-11-02, 02:48
Please send your logs also. It will help us understand if there is an
error during startup.

Hari

On Thu, Nov 1, 2012 at 6:44 PM, Siva Gudavalli
<[EMAIL PROTECTED]> wrote:
> hi Roshan,
>
> here is the config....
>
> agent.sources = avro-AppSrv-source
> agent.sinks = hdfs-Cluster-sink
> agent.channels = mem-channel
>
> # set channel for sources, sinks
> agent.sources.avro-AppSrv-source.channels = mem-channel
> agent.sources.hdfs-Cluster-sink.channel = mem-channel
>
> # set source properties
> agent.sources.avro-AppSrv-source.type = avro
> agent.sources.avro-AppSrv-source.bind = localhost
> agent.sources.avro-AppSrv-source.port = 41414
>
> # set channel properties
> agent.channels.mem-channel.type = memory
> agent.channels.mem-channel.capacity = 10000
> agent.channels.mem-channel.transactionCapacity = 10000
>
> # set sink properties
> agent.sinks.hdfs-Cluster-sink.type = hdfs
> agent.sinks.hdfs-Cluster-sink.hdfs.path = hdfs://localhost:54310/flume
>
> ________________________________________
> From: Roshan Naik [[EMAIL PROTECTED]]
> Sent: Friday, November 02, 2012 4:09 AM
> To: [EMAIL PROTECTED]
> Subject: Re: avro client fails to connect to flume
>
> Could you provide your flume configs ?
> -roshan
>
> On Thu, Nov 1, 2012 at 8:13 AM, Siva Gudavalli <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> hi,
>
> i am tryint to use avro client to source my log file to avro-source.
>
> flume-source -> avro source
> flume-channel -> memory
> flume-sink -> hdfs
>
> all of the services are up; but when i am calling avro client its failing connection to flume.
>
> please help!
>
> environment:
>
> hadoop 0.20.2
> flume 1.2
>
> logs from agent:
>
> 2012-11-01 20:03:49,977 INFO source.AvroSource: Avro source avro-AppSrv-source started.
> 2012-11-01 20:04:01,685 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] OPEN
> 2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] BOUND: /127.0.0.1:41414<http://127.0.0.1:41414>
> 2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] CONNECTED: /127.0.0.1:43370<http://127.0.0.1:43370>
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] DISCONNECTED
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] UNBOUND
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] CLOSED
>
> logs from avro client:
>
> Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh
> Info: Including Hadoop libraries found via (/usr/local/hadoop/bin/hadoop) for HDFS access
> + exec /usr/lib/jvm/java-7-openjdk-i386/bin/java -Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote -Dflume.root.logger=DEBUG,console -cp '/usr/local/flume/conf:/usr/local/flume/lib/*:/usr/local/flume/conf' -Djava.library.path=:/usr/local/hadoop/bin/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H localhost -p 41414 -F /usr/local/hbase/logs/hbase-hduser-master-172.20.204.119.log
> 2012-11-01 20:25:50,176 (main) [DEBUG - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] Batch size string = 5
> 2012-11-01 20:25:50,284 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 41414 }: RPC connection error
>         at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>         at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)