Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Sending avro data from other languages


Copy link to this message
-
Sending avro data from other languages
I'm playing around with making a standalone tail client in python(so
that I can access inode data) that tracks position in a file and then
sends it across avro to an avro sink.

However I'm having issues with the avro part of this and wondering if
anyone more familiar with it could help.

I took the flume.avdl file and converted it using "java -jar
~/Downloads/avro-tools-1.6.3.jar idl flume.avdl flume.avpr"

I then run it through a simple test program to see if its sending the
data correctly and it sends from the python client fine, but the sink
end OOM's because presumably the wire format is wrong:

2012-07-30 17:22:57,565 INFO ipc.NettyServer: [id: 0x5fc6e818,
/172.22.114.32:55671 => /172.28.19.112:41414] OPEN
2012-07-30 17:22:57,565 INFO ipc.NettyServer: [id: 0x5fc6e818,
/172.22.114.32:55671 => /172.28.19.112:41414] BOUND: /172.28.19.112:41414
2012-07-30 17:22:57,565 INFO ipc.NettyServer: [id: 0x5fc6e818,
/172.22.114.32:55671 => /172.28.19.112:41414] CONNECTED:
/172.22.114.32:55671
2012-07-30 17:22:57,646 WARN ipc.NettyServer: Unexpected exception from
downstream.
java.lang.OutOfMemoryError: Java heap space
         at java.util.ArrayList.<init>(ArrayList.java:112)
         at
org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154)
         at
org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131)
         at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
         at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
         at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
         at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
         at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
         at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
         at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:619)
2012-07-30 17:22:57,647 INFO ipc.NettyServer: [id: 0x5fc6e818,
/172.22.114.32:55671 :> /172.28.19.112:41414] DISCONNECTED
2012-07-30 17:22:57,647 INFO ipc.NettyServer: [id: 0x5fc6e818,
/172.22.114.32:55671 :> /172.28.19.112:41414] UNBOUND
2012-07-30 17:22:57,647 INFO ipc.NettyServer: [id: 0x5fc6e818,
/172.22.114.32:55671 :> /172.28.19.112:41414] CLOSED

I've dumped the test program and its output

http://pastebin.com/1DtXZyTu
http://pastebin.com/T9kaqKHY
+
Juhani Connolly 2012-08-02, 04:06
+
Brock Noland 2012-08-02, 14:01
+
Juhani Connolly 2012-08-03, 06:30
+
Brock Noland 2012-08-03, 12:49
+
Juhani Connolly 2012-08-06, 01:21
+
Hari Shreedharan 2012-08-06, 01:31
+
Juhani Connolly 2012-08-06, 04:26
+
Arvind Prabhakar 2012-08-02, 16:59
+
Brock Noland 2012-07-30, 13:31