Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> DBOutputFormat with one reducer


Copy link to this message
-
Re: DBOutputFormat with one reducer
On 05/25/2011 04:27 PM, Giridhar Addepalli wrote:
>
> Hi,
>
> We have MapReduce program which writes data to mysql database using
> DBOutputFormat.
>
> Our program has one reducer.
>
> I understand that all the inserts happen during the close() operation
> of the reducer.
>
> Is it gauranteed that this operation is atomic ? i.e; what happens if
> the writes fail in the middle of the operation.
>
> Does it mean that only partial number of rows get in to mysql database?
>
> What does it take to make the write operation atomic ?
>
> Any suggestions around our situation( alternative solutions ) are welcome.
>
> Thanks,
>
> Giridhar.
>
Sqoop is done for that kind of tasks.

Definition
=======  Sqoop is an open-source tool that allows users to extract data from a
relational database into Hadoop for further processing.
  This processing can be done with MapReduce programs or other
higher-level tools such  as Hive. When the final results of an analytic
pipeline are available, Sqoop can export
  these results back to the database for consumption by other clients.

Source code
=========  http://github.com/cloudera/sqoop

Regards

--
Marcos Luis Ortiz Valmaseda
  Software Engineer (Distributed Systems)
  http://uncubanitolinuxero.blogspot.com