Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Help determining exit code from flume avro-client


Copy link to this message
-
Re: Help determining exit code from flume avro-client
Hi Paul.

Thanks for reporting this. It does look like Flume does not return non-zero error code even if there was an exception. I filed https://issues.apache.org/jira/browse/FLUME-1670 to address this.

Thanks,
Hari

--
Hari Shreedharan
On Tuesday, October 30, 2012 at 11:25 AM, Paul Chavez wrote:

> I am working on getting the flume avro-client functionality working on Windows, and am currently stuck on how to determine if the file was sent successfully.
>  
> I used the blog post at http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to guide me and have built the 1.4.0 snapshot successfully. I can manually send files along to flume using the ''org.apache.flume.client.avro.AvroCLIClient" class.
>  
> The full command I am using is invoked via batch file with the following single line:
> "C:\Program Files (x86)\Java\jdk1.6.0_37\bin\java.exe" -Xmx20m -Dlog4j.configuration=file:///C:\apache-flume-1.4.0-SNAPSHOT\conf\log4j.properties -cp "C:\apache-flume-1.4.0-SNAPSHOT\lib\*" org.apache.flume.client.avro.AvroCLIClient -H 10.20.2.8 -p 6240 -F%1
>  
> My intention is to invoke from Powershell like so:
> .\flume.bat C:\IISLogs\W3SVC100\u_ex121029.log
>  
> This is working well and I am able to successfully send IIS and our app logs to flume and then onto HDFS without issues. Now I am trying to automate the process, and need to know if the operation failed for some reason. If I simply turn off the flume source I can trigger an error which is dumped to console like so (using flume.root.logger=DEBUG,console in the log4j file):
>  
>  
> 2012-10-30 11:14:22,185 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable t
> o open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: x.x.x.x, port: 6240 }: RPC connection error
>         at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>         at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>         at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>         at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>         at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
>         at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
> Caused by: java.io.IOException: Error connecting to /x.x.x.x:6240
>         at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>         at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>         at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>         at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>         ... 5 more
> Caused by: java.net.ConnectException: Connection refused: no further information
>         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
>         at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:
> 401)
>         at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipeli
> neSink.java:370)
>         at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
>  
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> 2012-10-30 11:14:22,185 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
>  
> Despite the ERROR message being logged to the console, java.exe still exits with code 0, indicating success. Is there a way to use the console appender to trigger java.exe to send a non-0 exit code if an error occurs so the rest of my automation tooling can detect the error? I can parse the output with Powershell and have it send a non-0 exit code itself, but looking for a more 'native' way to do this.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB