Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Reduce phase hangs


Copy link to this message
-
Re: Reduce phase hangs
It could be the order by since all the output records have to go through a
single reducer to get ordered.

Can you remove the order by and re-try the query?

You can read up more about what I am saying at
https://cwiki.apache.org/Hive/languagemanual-sortby.html

Mark

On Thu, Nov 8, 2012 at 5:23 AM, Robin Verlangen <[EMAIL PROTECTED]> wrote:

> Hmm, after looking in the job&task-tracker web interfaces it seemed that
> one of the new nodes was unable to connect to two of the others. This
> caused the copying of data to "hang" (in fact: timeout, on timeout, on
> timeout, ...).
>
> Best regards,
>
> Robin Verlangen
> *Software engineer*
> *
> *
> W http://www.robinverlangen.nl
> E [EMAIL PROTECTED]
>
> <http://goo.gl/Lt7BC>
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
>
>
> On Thu, Nov 8, 2012 at 2:15 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>
>> Maybe you need more heap or that nodes's health is not fine. Or maybe the
>> data block on that node is causing the code to go in a very long loop.
>> There can be several reasons. It's tough to say anything confidently
>> without knowing the real issue.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>>
>> On Thu, Nov 8, 2012 at 5:39 PM, Robin Verlangen <[EMAIL PROTECTED]> wrote:
>>
>>> It actually seems that only 1 of the 24 reducers hangs at the "copy"
>>> phase. Any solutions for this?
>>>
>>> Best regards,
>>>
>>> Robin Verlangen
>>> *Software engineer*
>>> *
>>> *
>>> W http://www.robinverlangen.nl
>>> E [EMAIL PROTECTED]
>>>
>>> <http://goo.gl/Lt7BC>
>>>
>>> Disclaimer: The information contained in this message and attachments is
>>> intended solely for the attention and use of the named addressee and may be
>>> confidential. If you are not the intended recipient, you are reminded that
>>> the information remains the property of the sender. You must not use,
>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>> received this message in error, please contact the sender immediately and
>>> irrevocably delete this message and any copies.
>>>
>>>
>>>
>>> On Thu, Nov 8, 2012 at 11:47 AM, Robin Verlangen <[EMAIL PROTECTED]> wrote:
>>>
>>>> Nothing special over there. Most of the jobs complete after quite some
>>>> time. However it makes no sense to me that it takes that long. Its probably
>>>> just a couple of hundreds of megabytes.
>>>>
>>>> Best regards,
>>>>
>>>> Robin Verlangen
>>>> *Software engineer*
>>>> *
>>>> *
>>>> W http://www.robinverlangen.nl
>>>> E [EMAIL PROTECTED]
>>>>
>>>> <http://goo.gl/Lt7BC>
>>>>
>>>> Disclaimer: The information contained in this message and attachments
>>>> is intended solely for the attention and use of the named addressee and may
>>>> be confidential. If you are not the intended recipient, you are reminded
>>>> that the information remains the property of the sender. You must not use,
>>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>>> received this message in error, please contact the sender immediately and
>>>> irrevocably delete this message and any copies.
>>>>
>>>>
>>>>
>>>> On Thu, Nov 8, 2012 at 11:43 AM, Mohammad Tariq <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hello Robin,
>>>>>
>>>>>         Look at the TaskTracker logs and see if you find anything
>>>>> interesting there.
>>>>>
>>>>> Regards,
>>>>>     Mohammad Tariq
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 8, 2012 at 4:03 PM, Robin Verlangen <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>> It seems that some of my jobs hang in the reduce phase for a very
>>>>>> long time (for example, days). Is there anything I could tweak on? The
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB