Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Writing Reducer output to database


Copy link to this message
-
Re: Writing Reducer output to database
Ted Yu 2011-02-04, 03:29
At least in cdh3b2, there are two DBOutputFormat.java:

./src/mapred/org/apache/hadoop/mapred/lib/db/DBOutputFormat.java
./src/mapred/org/apache/hadoop/mapreduce/lib/db/DBOutputFormat.java

You should be able to use the latter.

On Thu, Feb 3, 2011 at 2:45 PM, Adeel Qureshi <[EMAIL PROTECTED]>wrote:

> I had started a thread recently to ask questions about custom writable
> implementations which is basically similar to this .. but that was more of
> an understanding of the concept and here I wanted to ask my actual problem
> and get help on that.
>
> I want to be able to read text data line by line in my mapper ..
> create an instance of a custom writable class that holds some information
> parsed out of the line ..
> pass that custom writable along with its count to reducer
> reducer then simply need to insert every single entry into a database ..
>
> I am just trying to understand how to accomplish this. here is what I am
> thinking i need to do based on my little understanding of all this custom
> stuff
>
> 1. create a custom writable class that can hold my parsed records. in my
> mapper create a new instance of it using the text line and output the
> created instance
> 2. accept this custom writable in mapper
> 3. set reducer output to DBOutputFormat
>    I tried doing that and it seems like I am supposed to use JobConf class
> which is deprecated and the new configuration class where you are supposed
> to use the job object to set the input/output formats doesnt seems to work
> with DBOuputFormat .. doesnt this DBOutputFormat stuff works with hadoop
> new
> api
>
> 4. now in reducer I am confused wat to do .. i guess i need to convert my
> custom writable object to another custom dbwritable object .. that will
> then
> be written to the database .. any hints on how to accomplish this ..
>
> Sorry if the questions arent very clear .. I am just really confused about
> this stuff and it doesnt helps that there is literally NO useful
> information
> available anywhere on this writable and dbwritable stuff
>
> Thanks
> Adeel
>