Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Help determining exit code from flume avro-client

Paul Chavez 2012-10-30, 18:25
Hari Shreedharan 2012-10-30, 18:47
Copy link to this message
RE: Help determining exit code from flume avro-client
Thank you, I was unsure if that was desired behavior or not. I updated the class as a POC to add exit codes for each of those catch blocks and it works great.

From: Hari Shreedharan [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, October 30, 2012 11:47 AM
Subject: Re: Help determining exit code from flume avro-client

Hi Paul.

Thanks for reporting this. It does look like Flume does not return non-zero error code even if there was an exception. I filed https://issues.apache.org/jira/browse/FLUME-1670 to address this.


Hari Shreedharan
On Tuesday, October 30, 2012 at 11:25 AM, Paul Chavez wrote:

I am working on getting the flume avro-client functionality working on Windows, and am currently stuck on how to determine if the file was sent successfully.

I used the blog post at http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to guide me and have built the 1.4.0 snapshot successfully. I can manually send files along to flume using the ''org.apache.flume.client.avro.AvroCLIClient" class.

The full command I am using is invoked via batch file with the following single line:
"C:\Program Files (x86)\Java\jdk1.6.0_37\bin\java.exe" -Xmx20m -Dlog4j.configuration=file:///C:\apache-flume-1.4.0-SNAPSHOT\conf\log4j.properties -cp "C:\apache-flume-1.4.0-SNAPSHOT\lib\*" org.apache.flume.client.avro.AvroCLIClient -H -p 6240 -F%1

My intention is to invoke from Powershell like so:
.\flume.bat C:\IISLogs\W3SVC100\u_ex121029.log

This is working well and I am able to successfully send IIS and our app logs to flume and then onto HDFS without issues. Now I am trying to automate the process, and need to know if the operation failed for some reason. If I simply turn off the flume source I can trigger an error which is dumped to console like so (using flume.root.logger=DEBUG,console in the log4j file):
2012-10-30 11:14:22,185 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable t
o open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: x.x.x.x, port: 6240 }: RPC connection error
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
        at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
        at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
        at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
        at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
Caused by: java.io.IOException: Error connecting to /x.x.x.x:6240
        at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
        at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
        at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
        ... 5 more
Caused by: java.net.ConnectException: Connection refused: no further information
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipeli
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java2012-10-30 11:14:22,185 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting

Despite the ERROR message being logged to the console, java.exe still exits with code 0, indicating success. Is there a way to use the console appender to trigger java.exe to send a non-0 exit code if an error occurs so the rest of my automation tooling can detect the error? I can parse the output with Powershell and have it send a non-0 exit code itself, but looking for a more 'native' way to do this.

Paul Chavez
Hari Shreedharan 2012-10-30, 23:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB