Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - FW: WritableComparable value changing between Map and Reduce


+
Dave Shine 2012-06-26, 14:20
Copy link to this message
-
Re: FW: WritableComparable value changing between Map and Reduce
Stan Rosenberg 2012-06-26, 14:41
You are using the default timezone.  If the timezones differ, this could
account for the discrepancy. Based on your (de)serialization code, the
long value should be the same; its interpretation is different.

stan

On Tue, Jun 26, 2012 at 10:20 AM, Dave Shine <
[EMAIL PROTECTED]> wrote:

> After about a week of researching, logging, etc. I have finally discovered
> what is happening, but I have no idea why.****
>
> ** **
>
> I have created my own WritableComparable object so I can emit it as the
> key from my Mapper.  The object contains several Longs, one String, and one
> Date property.  The following code snippets are from the object****
>
> ** **
>
>                 private Date SummaryDate;****
>
> ** **
>
>                 /******
>
>                 * @return the summaryDate****
>
>                 */****
>
>                 public Date getSummaryDate() {****
>
>                                 return SummaryDate;****
>
>                 }****
>
> ** **
>
>                 /******
>
>                 * @param summaryDate the summaryDate to set****
>
>                 */****
>
>                 public void setSummaryDate(Date summaryDate) {****
>
>                                 Calendar cal = Calendar.getInstance();****
>
>                                 cal.setTime(summaryDate);****
>
>                                 cal.set(Calendar.HOUR, 0);****
>
>                                 cal.set(Calendar.MINUTE, 0);****
>
>                                 cal.set(Calendar.SECOND, 0);****
>
>                                 cal.set(Calendar.MILLISECOND, 0);****
>
>                                 cal.set(Calendar.AM_PM, Calendar.AM);****
>
>                                 SummaryDate = cal.getTime();****
>
>                 }****
>
> ** **
>
>                 @Override****
>
>                 public void readFields(DataInput arg0) throws IOException {
> ****
>
>                                 ....****
>
>                                 getSummaryDate().setTime(arg0.readLong());
> ****
>
>                 }****
>
> ** **
>
>                 @Override****
>
>                 public void write(DataOutput arg0) throws IOException {***
> *
>
>                                 ....****
>
>                                 arg0.writeLong(getSummaryDate().getTime());
> ****
>
>                 }****
>
> ** **
>
> The intent is for the Summary date to be always be as of midnight, thus
> the use of the Calendar object in the setSummaryDate() method.  ****
>
> ** **
>
> I have proven via logging that the Mapper is storing the correct value in
> the SummaryDate property, but sometimes the value received by the Reducer
> is the previous day.  Does anyone have any idea how this could happen?  **
> **
>
> ** **
>
> My only theory is precision on the Long where the epoch time is actually
> stored, that it somehow loses a tick and becomes 1 millisecond before
> midnight, then my code drops the time and the date portion is left with a
> date that is one day earlier.  Has anyone else seen anything like this?***
> *
>
> ** **
>
> I ready to go change my code to just store the date as a formatted
> string.  But I’d really like to know if this is a known Java or Hadoop
> problem. FWIW, I’m using CDH3U4.****
>
> ** **
>
> Thanks,****
>
> *Dave Shine*****
>
> Sr. Software Engineer****
>
> 321.939.5093 direct |  407.314.0122 mobile****
>
> ** **
>
> [image: cid:D34AFA33-EA7B-4B08-9DD4-2C8DFBE66338]****
>
> *CI Boost™ Clients*  *Outperform Online™  *www.ciboost.com****
>
> facebook platform | where-to-buy | product search engines | shopping
> engines****
>
> ** **
>
> ** **
>
> ** **
> ------------------------------
>
> The information contained in this email message is considered confidential
> and proprietary to the sender and is intended solely for review and use by
> the named recipient. Any unauthorized review, use or distribution is
> strictly prohibited. If you have received this message in error, please
> advise the sender by reply email and delete the message.****