Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Writing Reducer output to database

Copy link to this message
Re: Writing Reducer output to database
At least in cdh3b2, there are two DBOutputFormat.java:


You should be able to use the latter.

On Thu, Feb 3, 2011 at 2:45 PM, Adeel Qureshi <[EMAIL PROTECTED]>wrote:

> I had started a thread recently to ask questions about custom writable
> implementations which is basically similar to this .. but that was more of
> an understanding of the concept and here I wanted to ask my actual problem
> and get help on that.
> I want to be able to read text data line by line in my mapper ..
> create an instance of a custom writable class that holds some information
> parsed out of the line ..
> pass that custom writable along with its count to reducer
> reducer then simply need to insert every single entry into a database ..
> I am just trying to understand how to accomplish this. here is what I am
> thinking i need to do based on my little understanding of all this custom
> stuff
> 1. create a custom writable class that can hold my parsed records. in my
> mapper create a new instance of it using the text line and output the
> created instance
> 2. accept this custom writable in mapper
> 3. set reducer output to DBOutputFormat
>    I tried doing that and it seems like I am supposed to use JobConf class
> which is deprecated and the new configuration class where you are supposed
> to use the job object to set the input/output formats doesnt seems to work
> with DBOuputFormat .. doesnt this DBOutputFormat stuff works with hadoop
> new
> api
> 4. now in reducer I am confused wat to do .. i guess i need to convert my
> custom writable object to another custom dbwritable object .. that will
> then
> be written to the database .. any hints on how to accomplish this ..
> Sorry if the questions arent very clear .. I am just really confused about
> this stuff and it doesnt helps that there is literally NO useful
> information
> available anywhere on this writable and dbwritable stuff
> Thanks
> Adeel