Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> AVRO_EVENT problem


+
DeCarlo, Thom 2012-12-06, 14:59
+
Brock Noland 2012-12-06, 15:06
+
DeCarlo, Thom 2012-12-06, 15:09
+
Brock Noland 2012-12-06, 18:57
+
DeCarlo, Thom 2012-12-06, 19:29
Copy link to this message
-
Re: AVRO_EVENT problem
Hmm, looks like I missed something..Not familiar with this code.

On Thu, Dec 6, 2012 at 1:29 PM, DeCarlo, Thom <[EMAIL PROTECTED]> wrote:
> Hmmm... How does one define/check/set the schema? I lifted my configuration straight from the on-line documentation and it only mentions a schema in the JDBC Channel section.
>
> --
> Thom DeCarlo
>
>
> -----Original Message-----
> From: Brock Noland [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, December 06, 2012 1:57 PM
> To: [EMAIL PROTECTED]
> Subject: Re: AVRO_EVENT problem
>
> It seems to me like the object you are trying to write doesn't match
> the Schema? I'd check your convert() method.
>
> On Thu, Dec 6, 2012 at 9:09 AM, DeCarlo, Thom <[EMAIL PROTECTED]> wrote:
>> Oh, right... forgot the version. I'm using the latest (as of yesterday) from the git repository. It gets built as 1.4-SNAPSHOT.
>>
>> --
>> Thom DeCarlo
>>
>>
>> -----Original Message-----
>> From: Brock Noland [mailto:[EMAIL PROTECTED]]
>> Sent: Thursday, December 06, 2012 10:06 AM
>> To: [EMAIL PROTECTED]
>> Subject: Re: AVRO_EVENT problem
>>
>> Hi,
>>
>> Hopefully someone will be able to answer the AVRO issue, in order to
>> help them, what version of Flume are you running?
>>
>> Brock
>>
>> On Thu, Dec 6, 2012 at 8:59 AM, DeCarlo, Thom <[EMAIL PROTECTED]> wrote:
>>> Hi,
>>> I'm just getting started with flume, so I apologize if this is an already known problem.
>>>
>>> I'm trying to set up a FILE_ROLL sink that uses the AVRO_EVENT serializer. But, when I start the agent I get an exception thrown from within the AVRO DataFileWriter class. It is failing when trying to cast a java.util.ArrayList to java.util.Map.
>>>
>>> Can anyone tell me how this is supposed to work?
>>>
>>> OBTW, I'm running on Windows 7 (64-bit), and I've tried both the 32- and 64-bit versions of both Java 1.6.0_37 and 1.7.0_09, with the same results.
>>>
>>> Thanks,
>>> Thom
>>> --
>>> The console output looks like this:
>>>
>>> 2012-12-04 14:24:25,111 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.sink.RollingFileSink.start(RollingFileSink.java:135)] RollingFileSink fileSink-1 started.
>>> 2012-12-04 14:24:25,145 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:164)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:33333]
>>> 2012-12-04 14:24:25,593 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.channel.jdbc.impl.JdbcTransactionImpl.rollback(JdbcTransactionImpl.java:102)] Marking transaction for rollback
>>> 2012-12-04 14:24:25,594 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.channel.jdbc.impl.JdbcTransactionImpl.close(JdbcTransactionImpl.java:118)] Attempting transaction roll-back
>>> 2012-12-04 14:24:25,604 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
>>> org.apache.flume.EventDeliveryException: Failed to process transaction
>>>         at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:218)
>>>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>         at java.lang.Thread.run(Thread.java:722)
>>> Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.Map
>>>         at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263)
>>>         at org.apache.flume.serialization.AbstractAvroEventSerializer.write(AbstractAvroEventSerializer.java:108)
>>>         at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:195)
>>>         ... 3 more
>>> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.Map
>>>         at org.apache.avro.generic.GenericDatumWriter.getMapSize(GenericDatumWriter.java:174)
>>>         at org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter.java:159)

Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
+
Brock Noland 2012-12-06, 19:51
+
DeCarlo, Thom 2012-12-06, 20:37
+
Brock Noland 2012-12-06, 20:42
+
DeCarlo, Thom 2012-12-06, 20:48
+
DeCarlo, Thom 2012-12-07, 19:47
+
Brock Noland 2012-12-07, 19:50
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB