Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Thrift service + zklocks + time = death


+
Edward Capriolo 2012-04-11, 14:29
+
Edward Capriolo 2012-04-11, 14:43
Copy link to this message
-
Re: Thrift service + zklocks + time = death
Premise of the locking is that in well-behaved queries you will obtain a
lock at the start of the query and then relinquish it at the end of query.
Since, locks are lease-based, in badly-behaving queries after lease
expires, lock will be relinquished.
Are you are  NPE for same repeated query after couple of days it executing
fine or you have many different queries of which some acquire locks while
other dont? Do you see the failure on same kind of query every time or its
a different query every time. It will be useful to find out which queries
are resulting in this to debug this further.

Ashutosh
On Wed, Apr 11, 2012 at 07:43, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> BTW the problem is not tables being locked. On the server side
> acquireLocks is getting a null pointer.
>
> 32432495-OK
> 32432498:Error in acquireLocks: java.lang.NullPointerException
> 32432552-FAILED: Error in acquiring locks: locks on the underlying
> objects cannot be acquired. retry after some time
> 32432660-Hive history
> file=/tmp/hadoop/hive_job_log_hadoop_201204111032_691199436.txt
> 32432737-Hive history
> file=/tmp/hadoop/hive_job_log_hadoop_201204111032_1845073466.txt
>
>
> On Wed, Apr 11, 2012 at 10:29 AM, Edward Capriolo <[EMAIL PROTECTED]>
> wrote:
> > Hey all,
> >
> > Running 0.7.1
> > We are using a combination of hive-thrift+ zookeeper for locking. All
> > works well for a couple days until eventually we hit this condition.
> >
> > HiveServerException(message:Query returned non-zero code: 10, cause:
> > FAILED: Error in acquiring locks: locks on the underlying objects
> > cannot be acquired. retry after some time, errorCode:10,
> > SQLState:42000)
> >        at
> org.apache.hadoop.hive.service.ThriftHive$execute_result.read(ThriftHive.java:1494)
> >        at
> org.apache.hadoop.hive.service.ThriftHive$Client.recv_execute(ThriftHive.java:120)
> >        at
> org.apache.hadoop.hive.service.ThriftHive$Client.execute(ThriftHive.java:95)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >        at java.lang.reflect.Method.invoke(Method.java:597)
> >        at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
> >        at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
> >        at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
> >        at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
> >        at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
> >       XXX
> >       XXX
> >       XXX
> >        at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)
> >        at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
> >        at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
> >       XXX
> >       XXX
> >        at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)
> >        at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
> >        at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
> >        at core.ScriptTemplate.run(ScriptTemplate.groovy:28)
> >        at core.ScriptTemplate$run.call(Unknown Source)
> >        at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
> >        at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
> >        at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
+
Edward Capriolo 2012-04-11, 15:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB