Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >>


just add more, continue the above thread:

  protected synchronized long getEstimatedTotalMapOutputSize()  {
    if(completedMapsUpdates < threshholdToUse) {
      return 0;
    } else {
      long inputSize = job.getInputLength() + job.desiredMaps();
      //add desiredMaps() so that randomwriter case doesn't blow up
      //the multiplication might lead to overflow, casting it with
      //double prevents it
      long estimate = Math.round(((double)inputSize *
          completedMapsOutputSize * 2.0)/completedMapsInputSize);
      if (LOG.isDebugEnabled()) {
        LOG.debug("estimate total map output will be " + estimate);
      }
      return estimate;
    }
  }
On Sun, Jun 2, 2013 at 12:34 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:

> This should be fixed in hadoop-1.1.2 stable release.
> if we determine completedMapsInputSize is zero, then job's map tasks MUST
> be zero, so the estimated output size is zero.
> below is the code:
>
>   long getEstimatedMapOutputSize() {
>     long estimate = 0L;
>     if (job.desiredMaps() > 0) {
>       estimate = getEstimatedTotalMapOutputSize()  / job.desiredMaps();
>     }
>     return estimate;
>   }
>
>
>
> On Sat, Jun 1, 2013 at 11:49 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> Does smell like a bug as that number you get is simply Long.MAX_VALUE,
>> or 8 exbibytes.
>>
>> Looking at the sources, this turns out to be a rather funny Java issue
>> (there's a divide by zero happening and [1] suggests Long.MAX_VALUE
>> return in such a case). I've logged a bug report for this at
>> https://issues.apache.org/jira/browse/MAPREDUCE-5288 with a
>> reproducible case.
>>
>> Does this happen consistently for you?
>>
>> [1]
>> http://docs.oracle.com/javase/6/docs/api/java/lang/Math.html#round(double)
>>
>> On Sat, Jun 1, 2013 at 7:27 PM, Lanati, Matteo <[EMAIL PROTECTED]>
>> wrote:
>> > Hi all,
>> >
>> > I stumbled upon this problem as well while trying to run the default
>> wordcount shipped with Hadoop 1.2.0. My testbed is made up of 2 virtual
>> machines: Debian 7, Oracle Java 7, 2 GB RAM, 25 GB hard disk. One node is
>> used as JT+NN, the other as TT+DN. Security is enabled. The input file is
>> about 600 kB and the error is
>> >
>> > 2013-06-01 12:22:51,999 WARN org.apache.hadoop.mapred.JobInProgress: No
>> room for map task. Node 10.156.120.49 has 22854692864 bytes free; but we
>> expect map to take 9223372036854775807
>> >
>> > The logfile is attached, together with the configuration files. The
>> version I'm using is
>> >
>> > Hadoop 1.2.0
>> > Subversion
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r
>> 1479473
>> > Compiled by hortonfo on Mon May  6 06:59:37 UTC 2013
>> > From source with checksum 2e0dac51ede113c1f2ca8e7d82fb3405
>> > This command was run using
>> /home/lu95jib/hadoop-exmpl/hadoop-1.2.0/hadoop-core-1.2.0.jar
>> >
>> > If I run the default configuration (i.e. no securty), then the job
>> succeeds.
>> >
>> > Is there something missing in how I set up my nodes? How is it possible
>> that the envisaged value for the needed space is so big?
>> >
>> > Thanks in advance.
>> >
>> > Matteo
>> >
>> >
>> >
>> >>Which version of Hadoop are you using. A quick search shows me a bug
>> >>https://issues.apache.org/jira/browse/HADOOP-5241 that seems to show
>> >>similar symptoms. However, that was fixed a long while ago.
>> >>
>> >>
>> >>On Sat, Mar 23, 2013 at 4:40 PM, Redwane belmaati cherkaoui <
>> >>[EMAIL PROTECTED]> wrote:
>> >>
>> >>> This the content of the jobtracker log file :
>> >>> 2013-03-23 12:06:48,912 INFO org.apache.hadoop.mapred.JobInProgress:
>> Input
>> >>> size for job job_201303231139_0001 = 6950001. Number of splits = 7
>> >>> 2013-03-23 12:06:48,925 INFO org.apache.hadoop.mapred.JobInProgress:
>> >>> tip:task_201303231139_0001_m_000000 has split on
>> >>> node:/default-rack/hadoop0.novalocal
>> >>> 2013-03-23 12:06:48,927 INFO org.apache.hadoop.mapred.JobInProgress:
>> >>> tip:task_201303231139_0001_m_000001 has split on
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB