Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Error: Too Many Fetch Failures


Copy link to this message
-
Re: Error: Too Many Fetch Failures


You are probably having a very low somaxconn parameter ( default centos has it at 128 , if I remember correctly). You can check the value under /proc/sys/net/core/somaxconn

Can you also check the value of ulimit -n? It could be  low.

Raj

>________________________________
> From: Ellis H. Wilson III <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Sent: Tuesday, June 19, 2012 12:32 PM
>Subject: Re: Error: Too Many Fetch Failures
>
>On 06/19/12 13:38, Vinod Kumar Vavilapalli wrote:
>>
>> Replies/more questions inline.
>>
>>
>>> I'm using Hadoop 0.23 on 50 machines, each connected with gigabit ethernet and each having solely a single hard disk.  I am getting the following error repeatably for the TeraSort benchmark.  TeraGen runs without error, but TeraSort runs predictably until this error pops up between 64% and 70% completion.  This doesn't occur for every execution of the benchmark, as about one out of four times that I run the benchmark it does run to completion (TeraValidate included).
>>
>>
>> How many containers are you running per node?
>
>Per my attached config files, I specify that yarn.nodemanager.resource.memory-mb = 3072, and the default /seems/ to be set at 1024MB for maps and reducers, so I have 3 containers running per node.  I have verified that this indeed is the case in the web client.  Three of these 1GB "slots" in the cluster appear to be occupied by something else during the execution of TeraSort, so I specify that TeraGen create .5TB using 441 maps (3waves * (50nodes * 3containerslots - 3occupiedslots)), and TeraSort to use 147 reducers.  This seems to give me the guarantees I had with Hadoop 1.0 that each node gets an equal number of reducers, and my job doesn't drag on due to straggler reducers.
>
>> Clearly maps are getting killed because of fetch failures. Can you look at the logs of the NodeManager where this particular map task ran. That may have logs related to why reducers are not able to fetch map-outputs. It is possible that because you have only one disk per node, some of these nodes have bad or unfunctional disks and thereby causing fetch failures.
>
>I will rerun and report the exact error messages from the NodeManagers.  Can you give me more exacting advice on collecting logs of this sort, for as I mentioned I'm new to doing so with the new version of Hadoop? I have been looking in /tmp/logs and hadoop/logs, but perhaps there is somewhere else to look as well?
>
>Last, I am certain this is not related to failing disks, as this exact error occurs at much higher frequencies when I run Hadoop on a NAS box, which is the core of my research at the moment.  Nevertheless, I posted to this list instead of Dev as this was on vanilla CentOS-5.5 machines using just the HDDs within each, and therefore should be a highly typical setup.  In particular, I see these errors coming from numerous nodes all at once, and the subset of nodes giving the problems are not repeatable from one run to the next, though the resulting error is.
>
>> If that is the case, either you can offline these nodes or bump up mapreduce.reduce.shuffle.maxfetchfailures to tolerate these failures, the default is 10. There are other some tweaks which I can tell if you can find more details from your logs.
>
>I'd prefer to not bump up maxfetchfailures, and would rather simply fix the issue that is causing the fetch to fail in the beginning.  This isn't a large cluster, having only 50 nodes, nor are the links (1gig) or storage capabilities (1 sata drive) great or strange relative to any normal installation.  I have to assume here that I've mis-configured something :(.
>
>Best,
>
>ellis
>
>
>