Hi Mohit,

I believe you are running into a form of https://issues.apache.org/jira/browse/AURORA-1596. Assuming it is the same cause, you might try to increase the lock timeout introduced in the linked issue.

As a much more viable solution, I would recommend to update to 0.19 though: Aurora does no longer use H2 as the backend for its internal storage. Besides eliminating the mentioned error, it should also lead to significant performance improvements.

Best regards,
Stephan

From: Mohit Jaggi <[EMAIL PROTECTED]>
Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Date: Friday, 22. December 2017 at 20:25
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Subject: "Error accessing PooledConnection. Connection is invalid.

Folks,
While running load tests on Aurora I see the following. What does it mean? This<https://pastebin.com/cy0CXgMR> is the new code I added (we discussed this a few weeks ago) to decline incoming update API calls when Aurora is slowing down(not triggered in this case as this is a get call). The following exceptions were seen earlier w/o my code change and appear unrelated to the change. I also saw these during previous outages and want to understand what is going on.

Mohit.

------- Exception -----

Dec 20 16:57:46 node64 aurora-scheduler[78639]: W1220 16:57:46.061 [qtp337673094-129204, LoggingInterceptor:106] Uncaught exception while handling getJobUpdateDetails(null, JobUpdateQuery(role:test-uber-xxx-2, key:null, jobKey:null, user:null, updateStatuses:[], offset:0, limit:0)) org.apache.ibatis.exceptions.PersistenceException:
Dec 20 16:57:46 node64 aurora-scheduler[78639]: ### Error rolling back transaction.  Cause: java.sql.SQLException: Error accessing PooledConnection. Connection is invalid.
Dec 20 16:57:46 node64 aurora-scheduler[78639]: ### Cause: java.sql.SQLException: Error accessing PooledConnection. Connection is invalid.
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.ibatis.session.defaults.DefaultSqlSession.rollback(DefaultSqlSession.java:244)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.ibatis.session.SqlSessionManager.rollback(SqlSessionManager.java:315)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:116)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:83)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.storage.log.LogStorage.read(LogStorage.java:562)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.read(CallOrderEnforcingStorage.java:113)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobUpdateDetails(ReadOnlySchedulerImpl.java:327)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobUpdateDetails(SchedulerThriftInterface.java:1133)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.aop.ThrottlingInterceptor.invoke(ThrottlingInterceptor.java:122)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:68)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:98)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobUpdateDetails.getResult(ReadOnlyScheduler.java:1156)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobUpdateDetails.getResult(ReadOnlyScheduler.java:1141)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at org.apache.aurora.scheduler.http.api.TContentAwareServlet.doPost(TContentAwareServlet.java:154)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
/**
 * Binding module for AOP-style decorations of the thrift API.
 */
public class AopModule extends AbstractModule {

  private static final Matcher<? super Class<?>> THRIFT_IFACE_MATCHER =
      Matchers.subclassesOf(AnnotatedAuroraAdmin.class)
          .and(Matchers.annotatedWith(DecoratedThrift.class));

  @Override
  protected void configure() {
    // Layer ordering:
    // APIVersion -> Log -> StatsExporter -> SchedulerThriftInterface

    // It's important for this interceptor to be registered first to ensure it's at the 'top' of
    // the stack and the standard message is always applied.
    bindThriftDecorator(new ServerInfoInterceptor());

    bindThriftDecorator(new LoggingInterceptor());
    bindThriftDecorator(new ThriftStatsExporterInterceptor());
    bindThriftDecorator(new ThrottlingInterceptor());
  }

  private void bindThriftDecorator(MethodInterceptor interceptor) {
    bindThriftDecorator(binder(), THRIFT_IFACE_MATCHER, interceptor);
  }

  @VisibleForTesting
  static void bindThriftDecorator(
      Binder binder,
      Matcher<? super Class<?>> classMatcher,
      MethodInterceptor interceptor) {

    binder.bindInterceptor
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB