Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Job counters limit exceeded exception


Copy link to this message
-
Re: Job counters limit exceeded exception
Hi,

These happens when operators are used in queries (Hive Operators). Hive creates 4 counters per operator, max upto 1000, plus a few additional counters like file read/write, partitions and tables. Hence the number of counter required is going to be dependent upon the query.

Using "EXPLAIN EXTENDED" and "grep -ri operators | wc -l" print out the used numbers of operators. Use this value to tweak the MR settings carefully.

Praveen has a good explanation 'bout counters online:
http://www.thecloudavenue.com/2011/12/limiting-usage-counters-in-hadoop.html

Rule of thumb for Hive:
count of operators * 4 + n (n for file ops and other stuff).

cheers,
 Alex
On Jan 2, 2013, at 10:35 AM, Krishna Rao <[EMAIL PROTECTED]> wrote:

> A particular query that I run fails with the following error:
>
> ***
> Job 18: Map: 2  Reduce: 1   Cumulative CPU: 3.67 sec   HDFS Read: 0 HDFS
> Write: 0 SUCCESS
> Exception in thread "main"
> org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 121 max=120
> ...
> ***
>
> Googling suggests that I should increase "mapreduce.job.counters.limit".
> And that the number of counters a job uses
> has an effect on the memory used by the JobTracker, so I shouldn't increase
> this number too high.
>
> Is there a rule of thumb for what this number should be as a function of
> JobTracker memory? That is should I be cautious and
> increase by 5 at a time, or could I just double it?
>
> Cheers,
>
> Krishna

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB