Folks,
While running load tests on Aurora I see the following. What does it mean?
This <https://pastebin.com/cy0CXgMR> is the new code I added (we discussed
this a few weeks ago) to decline incoming update API calls when Aurora is
slowing down(not triggered in this case as this is a get call). The
following exceptions were seen earlier w/o my code change and appear
unrelated to the change. I also saw these during previous outages and want
to understand what is going on.

Mohit.

------- Exception -----

Dec 20 16:57:46 node64 aurora-scheduler[78639]: W1220 16:57:46.061
[qtp337673094-129204, LoggingInterceptor:106] Uncaught exception while
handling getJobUpdateDetails(null, JobUpdateQuery(role:test-uber-xxx-2,
key:null, jobKey:null, user:null, updateStatuses:[], offset:0, limit:0))
org.apache.ibatis.exceptions.PersistenceException:
Dec 20 16:57:46 node64 aurora-scheduler[78639]: ### Error rolling back
transaction.  Cause: java.sql.SQLException: Error accessing
PooledConnection. Connection is invalid.
Dec 20 16:57:46 node64 aurora-scheduler[78639]: ### Cause:
java.sql.SQLException: Error accessing PooledConnection. Connection is
invalid.
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.ibatis.session.defaults.DefaultSqlSession.rollback(DefaultSqlSession.java:244)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.ibatis.session.SqlSessionManager.rollback(SqlSessionManager.java:315)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:116)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:83)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.scheduler.storage.log.LogStorage.read(LogStorage.java:562)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.read(CallOrderEnforcingStorage.java:113)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getJobUpdateDetails(ReadOnlySchedulerImpl.java:327)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getJobUpdateDetails(SchedulerThriftInterface.java:1133)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.scheduler.thrift.aop.ThrottlingInterceptor.invoke(ThrottlingInterceptor.java:122)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:68)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:98)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobUpdateDetails.getResult(ReadOnlyScheduler.java:1156)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.gen.ReadOnlyScheduler$Processor$getJobUpdateDetails.getResult(ReadOnlyScheduler.java:1141)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
org.apache.aurora.scheduler.http.api.TContentAwareServlet.doPost(TContentAwareServlet.java:154)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
Dec 20 16:57:46 node64 aurora-scheduler[78639]: #011at
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

<https://pastebin.com/cy0CXgMR> at the end -----

/**
 * Binding module for AOP-style decorations of the thrift API.
 */
public class AopModule extends AbstractModule {

  private static final Matcher<? super Class<?>> THRIFT_IFACE_MATCHER =
      Matchers.subclassesOf(AnnotatedAuroraAdmin.class)
          .and(Matchers.annotatedWith(DecoratedThrift.class));

  @Override
  protected void configure() {
    // Layer ordering:
    // APIVersion -> Log -> StatsExporter -> SchedulerThriftInterface

    // It's important for this interceptor to be registered first to
ensure it's at the 'top' of
    // the stack and the standard message is always applied.
    bindThriftDecorator(new ServerInfoInterceptor());

    bindThriftDecorator(new LoggingInterceptor());
    bindThriftDecorator(new ThriftStatsExporterInterceptor());
    bindThriftDecorator(new ThrottlingInterceptor());
  }

  private void bindThriftDecorator(MethodInterceptor interceptor) {
    bindThriftDecorator(binder(), THRIFT_IFACE_MATCHER, interceptor);
  }

  @VisibleForTesting
  static void bindThriftDecorator(
      Binder binder,
      Matcher<? super Class<?>> classMatcher,
      MethodInterceptor interceptor) {

    binder.bindInterceptor(
        classMatcher,
        Matchers.returns(Matchers.subclassesOf(Response.class)),
        interceptor);
    binder.requestInjection(interceptor);
  }
}