Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Exceeded max jobconf size


+
Prateek Baranwal 2013-12-18, 16:55
Copy link to this message
-
Re: Exceeded max jobconf size
>> Pig version 0.10.1 (Cannot upgrade to 0.12)

Can you re-build your Pig from source tarball after applying this patch?
https://issues.apache.org/jira/browse/PIG-3017

If this is not possible, I can't think of any workaround other than making
your script smaller.

On Wed, Dec 18, 2013 at 8:55 AM, Prateek Baranwal <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have a python embedded script which works on transposing an year's worth
> by day columns.
> In order to do that python generates a lot of if else making the script
> 1600 lines long.
>
> On executing the script I land up with
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> java.io.IOException: Exceeded max jobconf size: 7698163 limit: 5242880
>         at
> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3734)
>         at
> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3698)
>         at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
> Caused by: java.io.IOException: Exceeded max jobconf size: 7698163 limit:
> 5242880
>         at
> org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:408)
>         at
> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3732)
>         ... 11 more
>
> I cannot change the job configuration size > 5m as I work from an edge
> node on a production server which I don't have access to make
> administrative changes.
> Has anyone faced this issue earlier ? Please suggest any work around or
> diagnostics I can do. :(
>
> Pig version 0.10.1 (Cannot upgrade to 0.12)
> Hadoop HDP 1.3
>
> I am also using joda-time and piggybank jar
>
> -Prateek
> Disclaimer: http://www.mu-sigma.com/disclaimer.html
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB