Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Writing Reducer output to database


+
Adeel Qureshi 2011-02-03, 22:45
+
Ted Yu 2011-02-04, 03:29
+
Adeel Qureshi 2011-02-04, 20:00
+
Adeel Qureshi 2011-02-06, 16:12
+
Ted Yu 2011-02-06, 17:00
+
Adeel Qureshi 2011-02-06, 23:28
Copy link to this message
-
Re: Writing Reducer output to database
Have you noticed this method in DBConfiguration ?
  public static void configureDB(Configuration conf, String driverClass,
      String dbUrl, String userName, String passwd) {

DBCountPageView has this call:
    DBConfiguration.configureDB(job, driverClassName, url);
On Sun, Feb 6, 2011 at 3:28 PM, Adeel Qureshi <[EMAIL PROTECTED]>wrote:

> Thanks Ted - That makes sense .. If I want the mapper to pass around a
> custom object as Key .. I need to provide the compareTo method = implement
> WritableComparable instead of Writable .. I did that and map stage is
> working fine now .. reducer is throwing an error now .. I am assuming its
> not able to communicate with database since its an IO exception but it
> doesnt provides much useful information ..
>
> 11/02/06 17:26:51 INFO mapred.JobClient: Task Id :
> attempt_201102061204_0004_r_000000_0, Status : FAILED
> java.io.IOException
>        at
>
> org.apache.hadoop.mapreduce.lib.db.DBOutputFormat.getRecordWriter(DBOutputFormat.java:180)
>        at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:557)
>        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>        at org.apache.hadoop.mapred.Child.main(Child.java:211)
>
>
> On Sun, Feb 6, 2011 at 11:00 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > I think you have looked at
> > src/examples/org/apache/hadoop/examples/DBCountPageView.java
> > where:
> >    job.setMapOutputKeyClass(Text.class);
> >
> > and:
> > public class Text extends BinaryComparable
> >    implements WritableComparable<BinaryComparable> {
> > ...
> >  static {
> >    // register this comparator
> >    WritableComparator.define(Text.class, new Comparator());
> >  }
> >
> > Modify your LogRecord accordingly.
> >
> > On Fri, Feb 4, 2011 at 12:00 PM, Adeel Qureshi <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Thanks - I switched to using the mapreduce version of dboutputformat
> and
> > > things look a little better but I am getting a ClassCastException ..
> > >
> > > here is my writable class
> > > public class LogRecord implements Writable, DBWritable {
> > >    private long timestamp;
> > >    private String userId;
> > >    private String action;
> > >
> > >    public LogRecord() {
> > >    }
> > >
> > >    public LogRecord(long timestamp, String userId, String action,
> > >            String pageType, String pageName, String attrPath, String
> > > attrName,
> > >            String forEntity, String forEntityInfo, long rendTime) {
> > >        this.timestamp = timestamp;
> > >        this.userId = userId;
> > >        this.action = action;
> > >    }
> > >
> > >    public void clearFields(){
> > >        this.timestamp = 0;
> > >        this.userId = "";
> > >        this.action = "";
> > >    }
> > >
> > >    @Override
> > >    public int hashCode() {
> > >        final int prime = 31;
> > >        int result = 1;
> > >        result = prime * result + ((action == null) ? 0 :
> > > action.hashCode());
> > >        result = prime * result + (int) (timestamp ^ (timestamp >>>
> 32));
> > >        result = prime * result + ((userId == null) ? 0 :
> > > userId.hashCode());
> > >        return result;
> > >    }
> > >
> > >    @Override
> > >    public boolean equals(Object obj) {
> > >        if (this == obj)
> > >            return true;
> > >        if (obj == null)
> > >            return false;
> > >        if (getClass() != obj.getClass())
> > >            return false;
> > >        LogRecord other = (LogRecord) obj;
> > >        if (action == null) {
> > >            if (other.action != null)
> > >                return false;
> > >        } else if (!action.equals(other.action))
> > >            return false;
> > >        if (timestamp != other.timestamp)
+
Adeel Qureshi 2011-02-07, 14:16
+
adeelmahmood 2011-02-04, 00:34
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB