Hadoop, mail # user - Re: Map Reduce jobs taking a long time at the end

Map Reduce jobs taking a long time at the end

Jay Whittaker 2012-12-04, 12:27

Hey,

We are running Map reduce jobs against a 12 machine hbase cluster and
for a long time they took approx 30 mins to return a result against ~95
million rows. Without any major changes to the data or any upgrade of
hbase/hadoop they now seem to be taking about 4 hours. and the logs are
full of

2012-12-04 13:33:15,602 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 72 6f 75
67 68 74
...
2012-12-04 13:45:17,134 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 72 70
6c 65 64 65 73 69 67 6e 73 65 72 76 69 63 65 73
...
2012-12-04 13:46:11,515 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 73 68
74 6f 74 61 6c 6b 2d 6f 6e 6c 69 6e 65

I presume the 0% is percent complete but I'm not sure as to why the time
to complete has now jumped massively. Ganglia shows no major load on the
nodes in question so I don't think it's that.

What steps should I be taking to try troubleshoot the problem?

Regards,

Jay

Re: Map Reduce jobs taking a long time at the end

ac@...) 2012-12-04, 13:59

Hi,

Have you also checked .out file of the tasktracker in logs? It could contain some useful information for the issue.

Thanks
ac
On 4 Dec 2012, at 8:27 PM, Jay Whittaker wrote:

> Hey,
>
> We are running Map reduce jobs against a 12 machine hbase cluster and
> for a long time they took approx 30 mins to return a result against ~95
> million rows. Without any major changes to the data or any upgrade of
> hbase/hadoop they now seem to be taking about 4 hours. and the logs are
> full of
>
> 2012-12-04 13:33:15,602 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 72 6f 75
> 67 68 74
> ...
> 2012-12-04 13:45:17,134 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 72 70
> 6c 65 64 65 73 69 67 6e 73 65 72 76 69 63 65 73
> ...
> 2012-12-04 13:46:11,515 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 73 68
> 74 6f 74 61 6c 6b 2d 6f 6e 6c 69 6e 65
>
> I presume the 0% is percent complete but I'm not sure as to why the time
> to complete has now jumped massively. Ganglia shows no major load on the
> nodes in question so I don't think it's that.
>
> What steps should I be taking to try troubleshoot the problem?
>
> Regards,
>
> Jay

Re: Map Reduce jobs taking a long time at the end

Jay Whittaker 2012-12-05, 15:47

Hey Ac,

The logs I copied were from the .out files while a job was running. I
thought that would be the best way to get a good idea of what was
happening.

Cheers,

Jay

On Tue, 2012-12-04 at 21:59 +0800, [EMAIL PROTECTED] wrote:
> Hi,
>
> Have you also checked .out file of the tasktracker in logs? It could contain some useful information for the issue.
>
> Thanks
> ac
>
>
> On 4 Dec 2012, at 8:27 PM, Jay Whittaker wrote:
>
> > Hey,
> >
> > We are running Map reduce jobs against a 12 machine hbase cluster and
> > for a long time they took approx 30 mins to return a result against ~95
> > million rows. Without any major changes to the data or any upgrade of
> > hbase/hadoop they now seem to be taking about 4 hours. and the logs are
> > full of
> >
> > 2012-12-04 13:33:15,602 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 72 6f 75
> > 67 68 74
> > ...
> > 2012-12-04 13:45:17,134 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 72 70
> > 6c 65 64 65 73 69 67 6e 73 65 72 76 69 63 65 73
> > ...
> > 2012-12-04 13:46:11,515 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 73 68
> > 74 6f 74 61 6c 6b 2d 6f 6e 6c 69 6e 65
> >
> > I presume the 0% is percent complete but I'm not sure as to why the time
> > to complete has now jumped massively. Ganglia shows no major load on the
> > nodes in question so I don't think it's that.
> >
> > What steps should I be taking to try troubleshoot the problem?
> >
> > Regards,
> >
> > Jay
>

Re: Map Reduce jobs taking a long time at the end

in.abdul 2012-12-05, 16:46

Hi jay..
  Are you trying to do M-R on HBase Table ?
Thanks and regards
Syed Abdul Kather
   Thanks and Regards,
   S SYED ABDUL KATHER

On Wed, Dec 5, 2012 at 9:53 PM, Jay Whittaker [via Lucene] <
ml-node+[EMAIL PROTECTED]> wrote:

> Hey Ac,
>
> The logs I copied were from the .out files while a job was running. I
> thought that would be the best way to get a good idea of what was
> happening.
>
> Cheers,
>
> Jay
>
> On Tue, 2012-12-04 at 21:59 +0800, [hidden email]<http://user/SendEmail.jtp?type=node&node=4024496&i=0>wrote:
>
> > Hi,
> >
> > Have you also checked .out file of the tasktracker in logs? It could
> contain some useful information for the issue.
> >
> > Thanks
> > ac
> >
> >
> > On 4 Dec 2012, at 8:27 PM, Jay Whittaker wrote:
> >
> > > Hey,
> > >
> > > We are running Map reduce jobs against a 12 machine hbase cluster and
> > > for a long time they took approx 30 mins to return a result against
> ~95
> > > million rows. Without any major changes to the data or any upgrade of
> > > hbase/hadoop they now seem to be taking about 4 hours. and the logs
> are
> > > full of
> > >
> > > 2012-12-04 13:33:15,602 INFO org.apache.hadoop.mapred.TaskTracker:
> > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 72 6f 75
> > > 67 68 74
> > > ...
> > > 2012-12-04 13:45:17,134 INFO org.apache.hadoop.mapred.TaskTracker:
> > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 72 70
> > > 6c 65 64 65 73 69 67 6e 73 65 72 76 69 63 65 73
> > > ...
> > > 2012-12-04 13:46:11,515 INFO org.apache.hadoop.mapred.TaskTracker:
> > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 73 68
> > > 74 6f 74 61 6c 6b 2d 6f 6e 6c 69 6e 65
> > >
> > > I presume the 0% is percent complete but I'm not sure as to why the
> time
> > > to complete has now jumped massively. Ganglia shows no major load on
> the
> > > nodes in question so I don't think it's that.
> > >
> > > What steps should I be taking to try troubleshoot the problem?
> > >
> > > Regards,
> > >
> > > Jay
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Map-Reduce-jobs-taking-a-long-time-at-the-end-tp4024231p4024496.html
> To start a new topic under Hadoop lucene-users, email
> ml-node+[EMAIL PROTECTED]
> To unsubscribe from Lucene, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=472066&code=aW4uYWJkdWxAZ21haWwuY29tfDQ3MjA2NnwxMDczOTUyNDEw>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
-----
THANKS AND REGARDS,
SYED ABDUL KATHER
--
View this message in context: http://lucene.472066.n3.nabble.com/Map-Reduce-jobs-taking-a-long-time-at-the-end-tp4024231p4024515.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Re: Map Reduce jobs taking a long time at the end

Jay Whittaker 2012-12-06, 12:42

Yeah, it's against a ~95million row table in hbase.

It takes about 30 mins to get to 90% then about 3+ hours to get from 90%
to 100%

On Wed, 2012-12-05 at 08:46 -0800, in.abdul wrote:
> Hi jay..
> Are you trying to do M-R on HBase Table ?
>
>
> Thanks and regards
> Syed Abdul Kather
>
>
> Thanks and Regards,
> S SYED ABDUL KATHER
>
>
>
> On Wed, Dec 5, 2012 at 9:53 PM, Jay Whittaker [via Lucene] <
> ml-node+[EMAIL PROTECTED]> wrote:
>
> > Hey Ac,
> >
> > The logs I copied were from the .out files while a job was running. I
> > thought that would be the best way to get a good idea of what was
> > happening.
> >
> > Cheers,
> >
> > Jay
> >
> > On Tue, 2012-12-04 at 21:59 +0800, [hidden email]<http://user/SendEmail.jtp?type=node&node=4024496&i=0>wrote:
> >
> > > Hi,
> > >
> > > Have you also checked .out file of the tasktracker in logs? It could
> > contain some useful information for the issue.
> > >
> > > Thanks
> > > ac
> > >
> > >
> > > On 4 Dec 2012, at 8:27 PM, Jay Whittaker wrote:
> > >
> > > > Hey,
> > > >
> > > > We are running Map reduce jobs against a 12 machine hbase cluster and
> > > > for a long time they took approx 30 mins to return a result against
> > ~95
> > > > million rows. Without any major changes to the data or any upgrade of
> > > > hbase/hadoop they now seem to be taking about 4 hours. and the logs
> > are
> > > > full of
> > > >
> > > > 2012-12-04 13:33:15,602 INFO org.apache.hadoop.mapred.TaskTracker:
> > > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 72 6f 75
> > > > 67 68 74
> > > > ...
> > > > 2012-12-04 13:45:17,134 INFO org.apache.hadoop.mapred.TaskTracker:
> > > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 72 70
> > > > 6c 65 64 65 73 69 67 6e 73 65 72 76 69 63 65 73
> > > > ...
> > > > 2012-12-04 13:46:11,515 INFO org.apache.hadoop.mapred.TaskTracker:
> > > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 73 68
> > > > 74 6f 74 61 6c 6b 2d 6f 6e 6c 69 6e 65
> > > >
> > > > I presume the 0% is percent complete but I'm not sure as to why the
> > time
> > > > to complete has now jumped massively. Ganglia shows no major load on
> > the
> > > > nodes in question so I don't think it's that.
> > > >
> > > > What steps should I be taking to try troubleshoot the problem?
> > > >
> > > > Regards,
> > > >
> > > > Jay
> > >
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> > http://lucene.472066.n3.nabble.com/Map-Reduce-jobs-taking-a-long-time-at-the-end-tp4024231p4024496.html
> > To start a new topic under Hadoop lucene-users, email
> > ml-node+[EMAIL PROTECTED]
> > To unsubscribe from Lucene, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=472066&code=aW4uYWJkdWxAZ21haWwuY29tfDQ3MjA2NnwxMDczOTUyNDEw>
> > .
> > NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >
>
>
>
>
> -----
> THANKS AND REGARDS,
> SYED ABDUL KATHER
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Map-Reduce-jobs-taking-a-long-time-at-the-end-tp4024231p4024515.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.