Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> FW: WritableComparable value changing between Map and Reduce


+
Dave Shine 2012-06-26, 14:20
Copy link to this message
-
Re: FW: WritableComparable value changing between Map and Reduce
You are using the default timezone.  If the timezones differ, this could
account for the discrepancy. Based on your (de)serialization code, the
long value should be the same; its interpretation is different.

stan

On Tue, Jun 26, 2012 at 10:20 AM, Dave Shine <
[EMAIL PROTECTED]> wrote:

> After about a week of researching, logging, etc. I have finally discovered
> what is happening, but I have no idea why.****
>
> ** **
>
> I have created my own WritableComparable object so I can emit it as the
> key from my Mapper.  The object contains several Longs, one String, and one
> Date property.  The following code snippets are from the object****
>
> ** **
>
>                 private Date SummaryDate;****
>
> ** **
>
>                 /******
>
>                 * @return the summaryDate****
>
>                 */****
>
>                 public Date getSummaryDate() {****
>
>                                 return SummaryDate;****
>
>                 }****
>
> ** **
>
>                 /******
>
>                 * @param summaryDate the summaryDate to set****
>
>                 */****
>
>                 public void setSummaryDate(Date summaryDate) {****
>
>                                 Calendar cal = Calendar.getInstance();****
>
>                                 cal.setTime(summaryDate);****
>
>                                 cal.set(Calendar.HOUR, 0);****
>
>                                 cal.set(Calendar.MINUTE, 0);****
>
>                                 cal.set(Calendar.SECOND, 0);****
>
>                                 cal.set(Calendar.MILLISECOND, 0);****
>
>                                 cal.set(Calendar.AM_PM, Calendar.AM);****
>
>                                 SummaryDate = cal.getTime();****
>
>                 }****
>
> ** **
>
>                 @Override****
>
>                 public void readFields(DataInput arg0) throws IOException {
> ****
>
>                                 ....****
>
>                                 getSummaryDate().setTime(arg0.readLong());
> ****
>
>                 }****
>
> ** **
>
>                 @Override****
>
>                 public void write(DataOutput arg0) throws IOException {***
> *
>
>                                 ....****
>
>                                 arg0.writeLong(getSummaryDate().getTime());
> ****
>
>                 }****
>
> ** **
>
> The intent is for the Summary date to be always be as of midnight, thus
> the use of the Calendar object in the setSummaryDate() method.  ****
>
> ** **
>
> I have proven via logging that the Mapper is storing the correct value in
> the SummaryDate property, but sometimes the value received by the Reducer
> is the previous day.  Does anyone have any idea how this could happen?  **
> **
>
> ** **
>
> My only theory is precision on the Long where the epoch time is actually
> stored, that it somehow loses a tick and becomes 1 millisecond before
> midnight, then my code drops the time and the date portion is left with a
> date that is one day earlier.  Has anyone else seen anything like this?***
> *
>
> ** **
>
> I ready to go change my code to just store the date as a formatted
> string.  But I’d really like to know if this is a known Java or Hadoop
> problem. FWIW, I’m using CDH3U4.****
>
> ** **
>
> Thanks,****
>
> *Dave Shine*****
>
> Sr. Software Engineer****
>
> 321.939.5093 direct |  407.314.0122 mobile****
>
> ** **
>
> [image: cid:D34AFA33-EA7B-4B08-9DD4-2C8DFBE66338]****
>
> *CI Boost™ Clients*  *Outperform Online™  *www.ciboost.com****
>
> facebook platform | where-to-buy | product search engines | shopping
> engines****
>
> ** **
>
> ** **
>
> ** **
> ------------------------------
>
> The information contained in this email message is considered confidential
> and proprietary to the sender and is intended solely for review and use by
> the named recipient. Any unauthorized review, use or distribution is
> strictly prohibited. If you have received this message in error, please
> advise the sender by reply email and delete the message.****