Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Map Reduce jobs taking a long time at the end


+
Jay Whittaker 2012-12-04, 12:27
+
ac@...) 2012-12-04, 13:59
+
Jay Whittaker 2012-12-05, 15:47
+
in.abdul 2012-12-05, 16:46
Copy link to this message
-
Re: Map Reduce jobs taking a long time at the end
Yeah, it's against a ~95million row table in hbase.

It takes about 30 mins to get to 90% then about 3+ hours to get from 90%
to 100%

On Wed, 2012-12-05 at 08:46 -0800, in.abdul wrote:
> Hi jay..
>   Are you trying to do M-R on HBase Table ?
>
>
> Thanks and regards
> Syed Abdul Kather
>
>
>             Thanks and Regards,
>         S SYED ABDUL KATHER
>
>
>
> On Wed, Dec 5, 2012 at 9:53 PM, Jay Whittaker [via Lucene] <
> ml-node+[EMAIL PROTECTED]> wrote:
>
> > Hey Ac,
> >
> > The logs I copied were from the .out files while a job was running. I
> > thought that would be the best way to get a good idea of what was
> > happening.
> >
> > Cheers,
> >
> > Jay
> >
> > On Tue, 2012-12-04 at 21:59 +0800, [hidden email]<http://user/SendEmail.jtp?type=node&node=4024496&i=0>wrote:
> >
> > > Hi,
> > >
> > > Have you also checked .out file of the tasktracker in logs? It could
> > contain some useful information for the issue.
> > >
> > > Thanks
> > > ac
> > >
> > >
> > > On 4 Dec 2012, at 8:27 PM, Jay Whittaker wrote:
> > >
> > > > Hey,
> > > >
> > > > We are running Map reduce jobs against a 12 machine hbase cluster and
> > > > for a long time they took approx 30 mins to return a result against
> > ~95
> > > > million rows. Without any major changes to the data or any upgrade of
> > > > hbase/hadoop they now seem to be taking about 4 hours. and the logs
> > are
> > > > full of
> > > >
> > > > 2012-12-04 13:33:15,602 INFO org.apache.hadoop.mapred.TaskTracker:
> > > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 72 6f 75
> > > > 67 68 74
> > > > ...
> > > > 2012-12-04 13:45:17,134 INFO org.apache.hadoop.mapred.TaskTracker:
> > > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 72 70
> > > > 6c 65 64 65 73 69 67 6e 73 65 72 76 69 63 65 73
> > > > ...
> > > > 2012-12-04 13:46:11,515 INFO org.apache.hadoop.mapred.TaskTracker:
> > > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 73 68
> > > > 74 6f 74 61 6c 6b 2d 6f 6e 6c 69 6e 65
> > > >
> > > > I presume the 0% is percent complete but I'm not sure as to why the
> > time
> > > > to complete has now jumped massively. Ganglia shows no major load on
> > the
> > > > nodes in question so I don't think it's that.
> > > >
> > > > What steps should I be taking to try troubleshoot the problem?
> > > >
> > > > Regards,
> > > >
> > > > Jay
> > >
> >
> >
> > ------------------------------
> >  If you reply to this email, your message will be added to the discussion
> > below:
> >
> > http://lucene.472066.n3.nabble.com/Map-Reduce-jobs-taking-a-long-time-at-the-end-tp4024231p4024496.html
> >  To start a new topic under Hadoop lucene-users, email
> > ml-node+[EMAIL PROTECTED]
> > To unsubscribe from Lucene, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=472066&code=aW4uYWJkdWxAZ21haWwuY29tfDQ3MjA2NnwxMDczOTUyNDEw>
> > .
> > NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >
>
>
>
>
> -----
> THANKS AND REGARDS,
> SYED ABDUL KATHER
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Map-Reduce-jobs-taking-a-long-time-at-the-end-tp4024231p4024515.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.