Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Writing Reducer output to database


Copy link to this message
-
Re: Writing Reducer output to database
At least in cdh3b2, there are two DBOutputFormat.java:

./src/mapred/org/apache/hadoop/mapred/lib/db/DBOutputFormat.java
./src/mapred/org/apache/hadoop/mapreduce/lib/db/DBOutputFormat.java

You should be able to use the latter.

On Thu, Feb 3, 2011 at 2:45 PM, Adeel Qureshi <[EMAIL PROTECTED]>wrote:

> I had started a thread recently to ask questions about custom writable
> implementations which is basically similar to this .. but that was more of
> an understanding of the concept and here I wanted to ask my actual problem
> and get help on that.
>
> I want to be able to read text data line by line in my mapper ..
> create an instance of a custom writable class that holds some information
> parsed out of the line ..
> pass that custom writable along with its count to reducer
> reducer then simply need to insert every single entry into a database ..
>
> I am just trying to understand how to accomplish this. here is what I am
> thinking i need to do based on my little understanding of all this custom
> stuff
>
> 1. create a custom writable class that can hold my parsed records. in my
> mapper create a new instance of it using the text line and output the
> created instance
> 2. accept this custom writable in mapper
> 3. set reducer output to DBOutputFormat
>    I tried doing that and it seems like I am supposed to use JobConf class
> which is deprecated and the new configuration class where you are supposed
> to use the job object to set the input/output formats doesnt seems to work
> with DBOuputFormat .. doesnt this DBOutputFormat stuff works with hadoop
> new
> api
>
> 4. now in reducer I am confused wat to do .. i guess i need to convert my
> custom writable object to another custom dbwritable object .. that will
> then
> be written to the database .. any hints on how to accomplish this ..
>
> Sorry if the questions arent very clear .. I am just really confused about
> this stuff and it doesnt helps that there is literally NO useful
> information
> available anywhere on this writable and dbwritable stuff
>
> Thanks
> Adeel
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB