Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Writing Reducer output to database


+
Adeel Qureshi 2011-02-03, 22:45
+
Ted Yu 2011-02-04, 03:29
+
Adeel Qureshi 2011-02-04, 20:00
+
Adeel Qureshi 2011-02-06, 16:12
+
Ted Yu 2011-02-06, 17:00
+
Adeel Qureshi 2011-02-06, 23:28
Copy link to this message
-
Re: Writing Reducer output to database
Ted Yu 2011-02-07, 04:55
Have you noticed this method in DBConfiguration ?
  public static void configureDB(Configuration conf, String driverClass,
      String dbUrl, String userName, String passwd) {

DBCountPageView has this call:
    DBConfiguration.configureDB(job, driverClassName, url);
On Sun, Feb 6, 2011 at 3:28 PM, Adeel Qureshi <[EMAIL PROTECTED]>wrote:

> Thanks Ted - That makes sense .. If I want the mapper to pass around a
> custom object as Key .. I need to provide the compareTo method = implement
> WritableComparable instead of Writable .. I did that and map stage is
> working fine now .. reducer is throwing an error now .. I am assuming its
> not able to communicate with database since its an IO exception but it
> doesnt provides much useful information ..
>
> 11/02/06 17:26:51 INFO mapred.JobClient: Task Id :
> attempt_201102061204_0004_r_000000_0, Status : FAILED
> java.io.IOException
>        at
>
> org.apache.hadoop.mapreduce.lib.db.DBOutputFormat.getRecordWriter(DBOutputFormat.java:180)
>        at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:557)
>        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>        at org.apache.hadoop.mapred.Child.main(Child.java:211)
>
>
> On Sun, Feb 6, 2011 at 11:00 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > I think you have looked at
> > src/examples/org/apache/hadoop/examples/DBCountPageView.java
> > where:
> >    job.setMapOutputKeyClass(Text.class);
> >
> > and:
> > public class Text extends BinaryComparable
> >    implements WritableComparable<BinaryComparable> {
> > ...
> >  static {
> >    // register this comparator
> >    WritableComparator.define(Text.class, new Comparator());
> >  }
> >
> > Modify your LogRecord accordingly.
> >
> > On Fri, Feb 4, 2011 at 12:00 PM, Adeel Qureshi <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks - I switched to using the mapreduce version of dboutputformat
> and
> > > things look a little better but I am getting a ClassCastException ..
> > >
> > > here is my writable class
> > > public class LogRecord implements Writable, DBWritable {
> > >    private long timestamp;
> > >    private String userId;
> > >    private String action;
> > >
> > >    public LogRecord() {
> > >    }
> > >
> > >    public LogRecord(long timestamp, String userId, String action,
> > >            String pageType, String pageName, String attrPath, String
> > > attrName,
> > >            String forEntity, String forEntityInfo, long rendTime) {
> > >        this.timestamp = timestamp;
> > >        this.userId = userId;
> > >        this.action = action;
> > >    }
> > >
> > >    public void clearFields(){
> > >        this.timestamp = 0;
> > >        this.userId = "";
> > >        this.action = "";
> > >    }
> > >
> > >    @Override
> > >    public int hashCode() {
> > >        final int prime = 31;
> > >        int result = 1;
> > >        result = prime * result + ((action == null) ? 0 :
> > > action.hashCode());
> > >        result = prime * result + (int) (timestamp ^ (timestamp >>>
> 32));
> > >        result = prime * result + ((userId == null) ? 0 :
> > > userId.hashCode());
> > >        return result;
> > >    }
> > >
> > >    @Override
> > >    public boolean equals(Object obj) {
> > >        if (this == obj)
> > >            return true;
> > >        if (obj == null)
> > >            return false;
> > >        if (getClass() != obj.getClass())
> > >            return false;
> > >        LogRecord other = (LogRecord) obj;
> > >        if (action == null) {
> > >            if (other.action != null)
> > >                return false;
> > >        } else if (!action.equals(other.action))
> > >            return false;
> > >        if (timestamp != other.timestamp)
+
Adeel Qureshi 2011-02-07, 14:16
+
adeelmahmood 2011-02-04, 00:34