Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> AVRO_EVENT problem


Copy link to this message
-
AVRO_EVENT problem
Hi,
I'm just getting started with flume, so I apologize if this is an already known problem.

I'm trying to set up a FILE_ROLL sink that uses the AVRO_EVENT serializer. But, when I start the agent I get an exception thrown from within the AVRO DataFileWriter class. It is failing when trying to cast a java.util.ArrayList to java.util.Map.

Can anyone tell me how this is supposed to work?

OBTW, I'm running on Windows 7 (64-bit), and I've tried both the 32- and 64-bit versions of both Java 1.6.0_37 and 1.7.0_09, with the same results.

Thanks,
Thom
--
The console output looks like this:

2012-12-04 14:24:25,111 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.sink.RollingFileSink.start(RollingFileSink.java:135)] RollingFileSink fileSink-1 started.
2012-12-04 14:24:25,145 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:164)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:33333]
2012-12-04 14:24:25,593 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.channel.jdbc.impl.JdbcTransactionImpl.rollback(JdbcTransactionImpl.java:102)] Marking transaction for rollback
2012-12-04 14:24:25,594 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.jdbc.impl.JdbcTransactionImpl.close(JdbcTransactionImpl.java:118)] Attempting transaction roll-back
2012-12-04 14:24:25,604 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to process transaction
        at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:218)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.Map
        at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263)
        at org.apache.flume.serialization.AbstractAvroEventSerializer.write(AbstractAvroEventSerializer.java:108)
        at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:195)
        ... 3 more
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.Map
        at org.apache.avro.generic.GenericDatumWriter.getMapSize(GenericDatumWriter.java:174)
        at org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter.java:159)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:68)
        at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
        at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
        at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
        at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257)
        ... 5 more

--
My config file looks like this:

########################################
# Sources
########################################
agent1.sources = source1

# Describe/configure source1
agent1.sources.source1.type = netcat
agent1.sources.source1.bind = localhost
agent1.sources.source1.port = 33333
agent1.sources.source1.interceptors = inter1 inter2
agent1.sources.source1.interceptors.inter1.type = timestamp
agent1.sources.source1.interceptors.inter2.type = host
agent1.sources.source1.interceptors.inter2.hostHeader = hostname

########################################
# Channels
########################################
agent1.channels = channel1 jdbcChannel-1

# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000
agent1.channels.channel1.transactionCapacity = 100

# Add a channel which uses JDBC to buffer messages
agent1.channels.jdbcChannel-1.type = jdbc

########################################
# Sinks
########################################
agent1.sinks = sink1 fileSink-1

# Describe sink1, a simple console Logger Sink
agent1.sinks.sink1.type = logger

#Describe fileSink-1, which writes to a file
agent1.sinks.fileSink-1.type = FILE_ROLL
agent1.sinks.fileSink-1.sink.directory = c:/flume/logs
agent1.sinks.fileSink-1.sink.rollInterval = 3600
#agent1.sinks.fileSink-1.sink.serializer = org.apache.flume.serialization.HeaderAndBodyTextEventSerializer$Builder
agent1.sinks.fileSink-1.sink.serializer = AVRO_EVENT
agent1.sinks.fileSink-1.sink.serializer.compressionCodec = snappy

########################################
# Bindings
########################################
# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1 jdbcChannel-1
agent1.sinks.sink1.channel = channel1
agent1.sinks.fileSink-1.channel = jdbcChannel-1
Thom DeCarlo
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB