Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Mapping MySQL schema to Avro


Copy link to this message
-
Re: Mapping MySQL schema to Avro
Hey Bart,

Why not using Sqoop for trasforming data from/to hadoop to/from MySQL?
http://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html
see the parameter: -as-avrodatafile

Ruslan
On Sat, Nov 24, 2012 at 2:20 AM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> This is probably the wrong list for your question.
>
> And, no, I don't think that your conversion is correct.
>
> To me it looks like you have lots of values in a fmsession.  In the avro
> version, those values appear to be repeated in the fmsswitchvalues. That
> seems wrong.
>
>
> On Fri, Nov 23, 2012 at 5:12 AM, Bart Verwilst <[EMAIL PROTECTED]> wrote:
>
>> Hello!
>>
>> I'm currently writing an importer to import our MySQL data into hadoop (
>> as Avro files ). Attached you can find the schema i'm converting to Avro,
>> along with the corresponding Avro schema i would like to use for my
>> imported data. I was wondering if you guys could go over the schema and
>> determine if this is sane/optimal, and if not, how i should improve it.
>>
>> As a sidenote, i converted bigints to long, and had 1 occurrence of
>> double, which i also converted to long in the avro, not sure if that's the
>> correct type?
>>
>> Thanks in advance for your expert opinions! ;)
>>
>> Kind regards,
>>
>> Bart
>>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB