Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> AVRO_EVENT problem


Copy link to this message
-
RE: AVRO_EVENT problem
Oh, not quite as easy as that. I hit the Hadoop dependency in the FILE_CHANNEL, as mentioned back in November
(http://mail-archives.apache.org/mod_mbox/flume-user/201211.mbox/%3CE3E225C9E0CD4AC28A90C4E463D717E1%40cloudera.com%3E)

Now I need to work out how to resolve that on a Win7 platform.

--
Thom DeCarlo
[EMAIL PROTECTED]
(o) 703-983-2817
(m) 757-880-6559

-----Original Message-----
From: DeCarlo, Thom [mailto:[EMAIL PROTECTED]]
Sent: Thursday, December 06, 2012 3:48 PM
To: [EMAIL PROTECTED]
Subject: RE: AVRO_EVENT problem

Thanks for the info. I will skip the jdbc channel and stick with the file channel.

--
Thom DeCarlo

-----Original Message-----
From: Brock Noland [mailto:[EMAIL PROTECTED]]
Sent: Thursday, December 06, 2012 3:42 PM
To: [EMAIL PROTECTED]
Subject: Re: AVRO_EVENT problem

Hi,

I have created https://issues.apache.org/jira/browse/FLUME-1771 for this issue.

Regarding JDBCChannel. Personally I would use FileChannel. It should
be just as reliable and it's deployed at scale in a large variety of
use cases.

Brock

On Thu, Dec 6, 2012 at 2:37 PM, DeCarlo, Thom <[EMAIL PROTECTED]> wrote:
> Interesting. It looks like the JDBC Channel causes problems for the file_roller+avro_event_serializer sink. Everything works right after switching that sink to use the Memory channel. Also, the logger sink is perfectly happy to use the jdbc channel.
>
> After thinking about this some more (and looking at the file output using the avro_event serializer), maybe it doesn't make sense to use that serializer the way I've attempted. Unless the file is read later by an AVRO reader the file won't be useful because it contains non-printing characters, which probably mean something to AVRO, but not much at all to vi.
>
> I'm going to try using the Netcat source->JDBC channel->AVRO sink, which will send to an AVRO source->JDBC channel->file_roller sink and see if the jdbc will work in there. (My target system is going to need high reliability message passing and the jdbc channel sounds like it is the most robust of the available channels.)
> --
> Thom DeCarlo
>
>
> -----Original Message-----
> From: Brock Noland [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, December 06, 2012 2:52 PM
> To: [EMAIL PROTECTED]
> Subject: Re: AVRO_EVENT problem
>
> OK, I don't really understand how Avro is working here, but I think
> you should try FileChannel or maybe MemoryChannel for simplicity to
> see if that works. IE, I think the problem is JDBCChannel. Can you let
> me know how it turns out?
>
> On Thu, Dec 6, 2012 at 1:45 PM, Brock Noland <[EMAIL PROTECTED]> wrote:
>> Hmm, looks like I missed something..Not familiar with this code.
>>
>> On Thu, Dec 6, 2012 at 1:29 PM, DeCarlo, Thom <[EMAIL PROTECTED]> wrote:
>>> Hmmm... How does one define/check/set the schema? I lifted my configuration straight from the on-line documentation and it only mentions a schema in the JDBC Channel section.
>>>
>>> --
>>> Thom DeCarlo
>>>
>>>
>>> -----Original Message-----
>>> From: Brock Noland [mailto:[EMAIL PROTECTED]]
>>> Sent: Thursday, December 06, 2012 1:57 PM
>>> To: [EMAIL PROTECTED]
>>> Subject: Re: AVRO_EVENT problem
>>>
>>> It seems to me like the object you are trying to write doesn't match
>>> the Schema? I'd check your convert() method.
>>>
>>> On Thu, Dec 6, 2012 at 9:09 AM, DeCarlo, Thom <[EMAIL PROTECTED]> wrote:
>>>> Oh, right... forgot the version. I'm using the latest (as of yesterday) from the git repository. It gets built as 1.4-SNAPSHOT.
>>>>
>>>> --
>>>> Thom DeCarlo
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Brock Noland [mailto:[EMAIL PROTECTED]]
>>>> Sent: Thursday, December 06, 2012 10:06 AM
>>>> To: [EMAIL PROTECTED]
>>>> Subject: Re: AVRO_EVENT problem
>>>>
>>>> Hi,
>>>>
>>>> Hopefully someone will be able to answer the AVRO issue, in order to
>>>> help them, what version of Flume are you running?
>>>>
>>>> Brock
>>>>
>>>> On Thu, Dec 6, 2012 at 8:59 AM, DeCarlo, Thom <[EMAIL PROTECTED]> wrote:
>>>>> Hi,

Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/