Dan O.

  • Fishkill, NY
PUBLIC PROFILE

System down notes

Posted by Dan OConnor on January 13, 2014 3:35 PM EST
Dan O. photo
At about 1pm ET today I was asked to get things going. I’m just now wrapping up my analysis at 3pm.
 
Chaikinpowertools.com had an issue today whereby the final solution was to just restart Tomcat to get things working.
 
Here is what I observed leading up to the restart:
 
1. As a logged out user, the site worked just fine for all of the top-level tabs.
2. As a logged in user (using the hidden /login) I was able to login and access Connect's tabs just fine.
3. Whether I was logged in or not, as soon as I went to https://www.chaikinpowertools.com/portfolio-health-check.shtml there was a problem… the site hung for several seconds then my browser said that the site wasn't responding.
4. Unfortunately, now that my browser session was corrupt, none of the site pages would work. I got the same site not responding error for any page.
5. Going to "/logout" worked and I was able to use chaikinpowertools.com again, except for PHC.
6. On the API side, I tried accessing http://app.chaikinpowertools.com/CPTRestSecure/app/authenticate/getAuthorization?email=mrajkowski@concursive.com&password=test from the server and CPT gave me a Gateway 504 time-out error. Then Seth gave me his URL and it worked without a time-out.
7. I restarted Tomcat.
 
Some things to look at:
 
1) The Gateway 504s probably contributed to this, but why? The session between Concursive and CPTRestSecure was having some trouble, but does that drive the whole site? Possibly in the SecurityHook?
 
2) DEBUG mode is on but this generates a slowdown and lots of output. I had wished there were things logged in RegisterService, LoginService and PortfolioHealthCheckGetAllDataViewer so that I knew when something was being tried and then what the result was. I throw this out there so that if this happens again there will be more debug info by adding logger messages.
 
3) Database connection errors popped up in the CRM log (attached). Not sure if this is causing any issue between Connect and CRM. These should be reviewed and fixed. The log implies that there is a closed connection from the pool… maybe it was forced closed due to a lengthy operation. A lengthy operation will need to be declared, if so.
 
That’s all I could find.
 
Regards,
Matt
 
 
 
2014-01-13 12:40:32 ERROR com.concursive.crm.web.modules.api.actions.ProcessPacket - API error
java.lang.IllegalStateException: getAttribute: Session already invalidated
        at org.apache.catalina.session.StandardSession.getAttribute(StandardSession.java:1077)
        at org.apache.catalina.session.StandardSessionFacade.getAttribute(StandardSessionFacade.java:110)
        at com.concursive.crm.web.controller.actions.CRMModule.getConnection(Unknown Source)
        at com.concursive.crm.web.modules.api.actions.ProcessPacket.executeCommandDefault(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2810.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.concursive.crm.web.controller.servlets.ControllerServlet.service(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.concursive.crm.web.controller.filters.ClientTypeRequestFilter.doFilter(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864)
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665)
        at java.lang.Thread.run(Thread.java:662)
org.postgresql.util.PSQLException: This connection has been closed.
 
2014-01-13 12:46:32 DEBUG com.concursive.commons.web.mvc.servlets.ControllerServlet - Forward: /portal_page.jsp
org.postgresql.util.PSQLException: This connection has been closed.
        at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:822)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.prepareStatement(AbstractJdbc3Connection.java:273)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.prepareStatement(AbstractJdbc2Connection.java:301)
        at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.queryRecord(Unknown Source)
        at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.<init>(Unknown Source)
        at com.concursive.crm.web.modules.communications.actions.ProcessCampaignConfirmation.executeCommandDefault(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2667.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.concursive.crm.web.controller.servlets.ControllerServlet.service(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.concursive.crm.web.controller.filters.ClientTypeRequestFilter.doFilter(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864)
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665)
        at java.lang.Thread.run(Thread.java:662)
org.postgresql.util.PSQLException: This connection has been closed.
2014-01-13 12:46:32 WARN  com.concursive.commons.db.ConnectionPool - Connection has already been returned to pool
> No Action found, so doing nothing.
        at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:822)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.prepareStatement(AbstractJdbc3Connection.java:273)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.prepareStatement(AbstractJdbc2Connection.java:301)
        at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.queryRecord(Unknown Source)
        at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.<init>(Unknown Source)
        at com.concursive.crm.web.modules.communications.actions.ProcessCampaignConfirmation.executeCommandDefault(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2667.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.concursive.crm.web.controller.servlets.ControllerServlet.service(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.concursive.crm.web.controller.filters.ClientTypeRequestFilter.doFilter(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864)
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665)
        at java.lang.Thread.run(Thread.java:662)
org.postgresql.util.PSQLException: This connection has been closed.
2014-01-13 12:46:32 WARN  com.concursive.commons.db.ConnectionPool - Connection has already been returned to pool
> No Action found, so doing nothing.
        at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:822)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.prepareStatement(AbstractJdbc3Connection.java:273)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.prepareStatement(AbstractJdbc2Connection.java:301)
        at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.queryRecord(Unknown Source)
        at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.<init>(Unknown Source)
        at com.concursive.crm.web.modules.communications.actions.ProcessCampaignConfirmation.executeCommandDefault(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2667.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.concursive.crm.web.controller.servlets.ControllerServlet.service(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.concursive.crm.web.controller.filters.ClientTypeRequestFilter.doFilter(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864)
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665)
        at java.lang.Thread.run(Thread.java:662)
org.postgresql.util.PSQLException: This connection has been closed.
2014-01-13 12:46:33 WARN  com.concursive.commons.db.ConnectionPool - Connection has already been returned to pool
> No Action found, so doing nothing.
        at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:822)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.prepareStatement(AbstractJdbc3Connection.java:273)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.prepareStatement(AbstractJdbc2Connection.java:301)
        at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.queryRecord(Unknown Source)
        at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.<init>(Unknown Source)
        at com.concursive.crm.web.modules.communications.actions.ProcessCampaignConfirmation.executeCommandDefault(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2667.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.concursive.crm.web.controller.servlets.ControllerServlet.service(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.concursive.crm.web.controller.filters.ClientTypeRequestFilter.doFilter(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864) 
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665)
        at java.lang.Thread.run(Thread.java:662)
2014-01-13 12:46:33 WARN  com.concursive.commons.db.ConnectionPool - Connection has already been returned to pool
> No Action found, so doing nothing.
org.postgresql.util.PSQLException: This connection has been closed.
        at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:822)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.prepareStatement(AbstractJdbc3Connection.java:273)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.prepareStatement(AbstractJdbc2Connection.java:301)
        at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.queryRecord(Unknown Source)
        at com.concursive.crm.web.modules.communications.dao.ScheduledRecipient.<init>(Unknown Source)
        at com.concursive.crm.web.modules.communications.actions.ProcessCampaignConfirmation.executeCommandDefault(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor2667.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.concursive.crm.web.controller.servlets.ControllerServlet.service(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.concursive.crm.web.controller.filters.ClientTypeRequestFilter.doFilter(Unknown Source)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:864) 
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1665)
        at java.lang.Thread.run(Thread.java:662)
2014-01-13 12:46:33 WARN  com.concursive.commons.db.ConnectionPool - Connection has already been returned to pool
> No Action found, so doing nothing.
2014-01-13 12:46:36 DEBUG com.concursive.connect.web.filters.UrlRewriteFilter - requestedPath: /ProjectManagementFiles.do?command=Download&pid=2&fid=89632
 
I was closing some tabs and ran across these if they help. The gateway error occurred directly from the chaikin server and my local browser.Screen Shot 2014-01-13 at 3.20.23 PM.png
Screen Shot 2014-01-13 at 3.21.36 PM.png

There are no comments

Sign in to add your comment.

Recent Posts

System down notes
At about 1pm ET today I was asked to get things going. I’m just now wrapping up my analysis at 3pm....
read more

Go to blog