System down notes

Posted by Dan OConnor on January 13, 2014, 3:35 PM EST

At about 1pm ET today I was asked to get things going. I’m just now wrapping up my analysis at 3pm.

Chaikinpowertools.com had an issue today whereby the final solution was to just restart Tomcat to get things working.

Here is what I observed leading up to the restart:

1. As a logged out user, the site worked just fine for all of the top-level tabs.

2. As a logged in user (using the hidden /login) I was able to login and access Connect's tabs just fine.

3. Whether I was logged in or not, as soon as I went to https://www.chaikinpowertools.com/portfolio-health-check.shtml there was a problem… the site hung for several seconds then my browser said that the site wasn't responding.

4. Unfortunately, now that my browser session was corrupt, none of the site pages would work. I got the same site not responding error for any page.

5. Going to "/logout" worked and I was able to use chaikinpowertools.com again, except for PHC.

6. On the API side, I tried accessing http://app.chaikinpowertools.com/CPTRestSecure/app/authenticate/getAuthorization?email=mrajkowski@concursive.com&password=test from the server and CPT gave me a Gateway 504 time-out error. Then Seth gave me his URL and it worked without a time-out.

7. I restarted Tomcat.

8. The app fully works and so does the http://app.chaikinpowertools.com/CPTRestSecure/app/authenticate/getAuthorization?email=mrajkowski@concursive.com&password=test URL.

Some things to look at:

1) The Gateway 504s probably contributed to this, but why? The session between Concursive and CPTRestSecure was having some trouble, but does that drive the whole site? Possibly in the SecurityHook?

2) DEBUG mode is on but this generates a slowdown and lots of output. I had wished there were things logged in RegisterService, LoginService and PortfolioHealthCheckGetAllDataViewer so that I knew when something was being tried and then what the result was. I throw this out there so that if this happens again there will be more debug info by adding logger messages.

3) Database connection errors popped up in the CRM log (attached). Not sure if this is causing any issue between Connect and CRM. These should be reviewed and fixed. The log implies that there is a closed connection from the pool… maybe it was forced closed due to a lengthy operation. A lengthy operation will need to be declared, if so.

That’s all I could find.

Regards,

Matt

2014-01-13 12:40:32 ERROR com.concursive.crm.web.modules.api.actions.ProcessPacket - API error

java.lang.IllegalStateException: getAttribute: Session already invalidated

at org.apache.catalina.session.StandardSession.getAttribute(StandardSession.java:1077)

at org.apache.catalina.session.StandardSessionFacade.getAttribute(StandardSessionFacade.java:110)

at com.concursive.crm.web.controller.actions.CRMModule.getConnection(Unknown Source)

at com.concursive.crm.web.modules.api.actions.ProcessPacket.executeCommandDefault(Unknown Source)

at sun.reflect.GeneratedMethodAccessor2810.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at com.concursive.crm.web.controller.servlets.ControllerServlet.service(Unknown Source)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at com.concursive.crm.web.controller.filters.ClientTypeRequestFilter.doFilter(Unknown Source)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)

at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864)

at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)

at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665)

at java.lang.Thread.run(Thread.java:662)

org.postgresql.util.PSQLException: This connection has been closed.

2014-01-13 12:46:32 DEBUG com.concursive.commons.web.mvc.servlets.ControllerServlet - Forward: /portal_page.jsp

org.postgresql.util.PSQLException: This connection has been closed.

at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:822)

at org.postgresql.jdbc3.AbstractJdbc3Connection.prepareStatement(AbstractJdbc3Connection.java:273)

at org.postgresql.jdbc2.AbstractJdbc2Connection.prepareStatement(AbstractJdbc2Connection.java:301)

at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.queryRecord(Unknown Source)

at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.<init>(Unknown Source)

at com.concursive.crm.web.modules.communications.actions.ProcessCampaignConfirmation.executeCommandDefault(Unknown Source)

at sun.reflect.GeneratedMethodAccessor2667.invoke(Unknown Source)